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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

Over  the  past  decade,  the  U.S.  Department  of  Homeland  Security  (DHS)  has 
issued  specific  policy  guidance  documents  directing  the  establishment  of  perfonnance 
measures  that  assess  national  preparedness.  These  policy  documents  include  the  2003 
Homeland  Security  Policy  Directive  (HSPD)-8,  the  Post-Katrina  Emergency 
Management  Refonn  Act  of  2006  (PKEMRA),  the  2010  Modernization  Act,  and  most 
recently  the  2011  Presidential  Policy  Directive  (PPD)-8.  Performance  reporting 
requirements  under  these  federal  documents  include  developing  preparedness  plans  and 
strategies;  reporting  performance  metrics  and  measures;  and  identifying  the  risks  of 
specific  threats,  and  recognizing  the  vulnerabilities  and  prioritized  objectives  needed  to 
mitigate  those  risks. 

Numerous  efforts  to  comply  with  performance  measurement  requirements  have 
occurred  over  the  past  decade,  but  FEMA  has  yet  to  develop  a  system  that  satisfies  these 
directives.  As  a  result,  there  is  increased  scrutiny  over  the  expenditure  of  federal  grant 
funds  and  Congress  has  reduced  the  amount  of  federal  grant  dollars  to  SLTT  programs  by 
approximately  85  percent  over  a  ten-year  period.1  Additionally,  state,  local,  tribal  and 
territorial  (SLTT)  entities  are  consistently  identified  in  DHS  Office  of  Inspector  General 
audits  as  lacking  a  capability  to  track  perfonnance  indicators. 

The  purpose  of  the  inquiry  for  this  thesis  is  to  develop  a  model  for  collecting, 
analyzing,  and  reporting  on  the  performance  management  of  SLTT  jurisdictions 
receiving  DHS  block  grant  funding  for  preparedness  programs.  This  model  is  generic  to 
account  for  the  various  geopolitical  and  strategic  influences  among  the  jurisdictions  as 
well  as  the  relevant  natural  and  human-caused  hazards  and  threats  that  present  risks  and 
vulnerabilities  to  each  entity. 


1  Department  of  Homeland  Security,  Budget-in-Brief  Fiscal  Years  2004-2014  (Washington,  DC: 
Department  of  Homeland  Security),  http://www.dhs.gov/dhs-budget,  accessed  October  4,  2014. 


1 


B.  BACKGROUND 

In  the  early  1990s,  Congress  found  that  excessive  waste  and  inefficiency  in 
federal  programs  undermined  the  public’s  confidence  in  government,  which  reduced  the 
federal  government’s  ability  to  address  public  needs  adequately.  Congress  also 
detennined  the  perfonnance  management  of  federal  agencies  was  insufficient  to  set 
policy  and  inform  spending  decisions.2  The  outcome  of  this  limitation  negatively  affected 
the  efficiency  and  effectiveness  of  federal  programs.3  The  frustration  led  to  the 
development  of  the  Government  Performance  and  Results  Act  (GPRA),  which  obligated 
all  federal  agencies  to  develop  strategic  and  performance  plans  with  an  annual  reporting 
requirement.  Congress  updated  the  GPRA  under  the  GPRA  Modernization  Act 
(GPRAMA)  of  2010,  which  retained  most  of  the  original  requirements  with  added 
requirements,  processes,  and  reports.4 

In  keeping  with  GPRA  and  GPRAMA  obligations,  federal  law  and  Presidential 
Directives  established  specific  perfonnance  management  requirements  for  DHS.  In  2003, 
the  Homeland  Security  Presidential  Directive  (HSPD)-8  charged  DHS  “to  establish 
measurable  readiness  priorities  and  targets,  include  readiness  metrics,  and  develop  a 
system  for  assessing  the  nation’s  overall  preparedness.”5  Three  years  later,  the  Post- 
Katrina  Emergency  Management  Refonn  Act  (PKEMRA)  of  2006  required  all  national 
preparedness  plans  and  strategies  to  include  “updated,  clear,  and  quantifiable 
performance  metrics,  measures,  and  outcomes.”6  Finally,  in  2011,  Presidential  Policy 
Directive  (PPD)-8  stated  national  preparedness  “shall  be  informed  by  the  risk  of  specific 


2  Government  Performance  and  Results  Act  of  1993,  Pub.  L.  No.  103-62,  107  Stat.  285  (1993). 

3  Ibid. 

4  Government  Performance  and  Results  Act  (GPRA)  Modernization  Act  of  2010,  Pub.  L.  No.  11 1-352, 
124  Stat.  3866  (2011).' 

5  George  W.  Bush,  Homeland  Security  Presidential  Directive-8,  Directive  on  National  Preparedness 
(Washington,  DC:  The  White  House,  2003),  1823,  http://www.gpo.gov/fdsys/pkg/WCPD-2003-12- 
22/pdf7WCPD-2003  - 1 2-22-Pg  1 822  .pdf. 

6  Department  of  Homeland  Security’  Appropriations  Act  2007  (Post-Katrina  Emergency  Management 
Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1428  (2006). 
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threats  and  vulnerabilities,  taking  into  account  regional  variations  and  include  concrete, 
measurable,  and  prioritized  objectives  to  mitigate  that  risk.”7 

Over  the  years,  DHS  leadership  initiated  many  projects  to  develop  perfonnance 
management  systems  that  would  collect  infonnation  and  report  on  state  preparedness 
capabilities.  A  few  of  the  systems  evolved  through  several  iterations — and  there  are 
indications  of  significant  stakeholder  involvement  on  some — but  the  systems  never 
reached  full  implementation.  Notwithstanding  all  the  directives  and  the  many 
implementation  attempts,  the  significant  shortfall  in  achieving  a  national  preparedness 
assessment  is  the  lack  of  “clear,  objective,  and  quantifiable  capability  requirements  and 
performance  measures  that  are  needed  to  identify  capability  gaps.”8 

The  ability  to  implement  a  national  preparedness  assessment  remains  laden  with 
many  weighty  challenges.  First,  many  organizations  collect  data  on  emergency 
preparedness  “for  different  purposes,  in  many  different  fonns,  and  to  differing  degrees  of 
thoroughness,”9  because  the  responsibility  for  all-hazard  protection,  response,  and 
mitigation  flows  across  all  levels  of  government.  In  short,  the  number  of  federal 
organizations  involved  in  the  nation’s  preparedness  activities  makes  for  a  massive  and 
complex  enterprise.  This  problem  is  even  more  compounded  if  all  state,  local,  tribal,  and 
territorial  jurisdictions — along  with  the  various  public  and  private  entities — are 
appropriately  included  in  the  equation. 

Second,  DHS  is  still  a  new  organization  experiencing  growing  pains  as  it  matures. 
For  example,  when  DHS  was  created,  all  preparedness  programs  within  the  Federal 
Emergency  Management  Agency  (FEMA)  were  transferred  to  DHS’s  new  Directorate  for 
Preparedness.  Congress  reversed  the  reorganizational  change  three  years  later  under  the 


7  Barack  Obama,  Presidential  Policy  Directive  8:  National  Preparedness  (Washington,  DC:  The  White 
House,  2011),  2,  http://www.dhs.gov/presidential-policy-directive-8-nationalpreparedness. 

8  David  C.  Maurer,  National  Preparedness:  FEMA  Has  Made  Progress,  but  Additional  Steps  Are 
Needed  to  Improve  Grant  Management  and  Assess  Capabilities  (GAO  13-637T)  (Washington,  DC:  U.S. 
Government  Accountability  Office,  2013),  10,  http://www.gao.gov/assets/660/655392.pdf. 

9  Government  Accountability  Office  (GAO),  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop 
and  Implement  a  System  to  Assess  National  Preparedness  Capabilities  (GAO-1 1-5 1R)  (Washington,  DC: 
GAO,  2013),  31,  http://www.gao.gOv/assets/100/97160.pdf. 
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Post-Katrina  Emergency  Management  Reform  Act  (PKEMRA),10  returning  most  of  the 
preparedness  programs  back  to  FEMA.  The  change  in  agency  ownership  of  developed 
systems  caused  a  breakdown  in  program  management.  For  example,  two  programs 
affected  by  the  transfer  were  the  National  Exercise  Program  and  the  comprehensive 
assessment  system.  While  PKEMRA  became  effective  in  March  of  2007,  many  of  the 
staff  positions  within  the  new  offices  were  not  filled  until  November  of  2008.  Not  only 
did  the  program  lose  a  majority  of  its  professional  expertise,  there  was  a  significant  gap 
in  resources  to  manage  the  programs  for  well  over  a  year. 1 1 

Third,  in  addition  to  the  numerous  organizational  pennutations,  FEMA  is  in  a 
constant  state  of  flux.  The  agency’s  response  to  an  increasing  trend  in  the  number  of 
disasters  and  emergencies  across  the  nation  affects  their  ability  to  manage  programs  and 
shepherd  new  initiatives  due  to  the  strain  on  responding  resources.  The  recent  economic 
downturn  also  affected  progress  due  to  increased  budgetary  constraints  at  all  levels  of 
government,  which  can  reduce  staff  and  program  funding.  DHS’s  Office  of  the  Inspector 
General  indicated  concern  that  FEMA  does  not  have  “sufficient  staff  focused  on  planning 
and  preparedness  efforts.”12 

Finally,  DHS  has  faced  numerous  external  challenges  to  implementing  a 
performance  management  system.  One  such  challenge  is  data  reliability,  which  is 
dependent  on  self-reported  preparedness  data  and  assessments  from  state,  local,  and  tribal 
jurisdictions.  A  related  challenge  is  the  development  of  standardized  metrics.  FEMA 
identified  these  various  challenges  and  has  thus  far  unsuccessfully  attempted  to  mitigate 
them  through  development  of  standardized,  web-based  data  collection  tools.13 

The  ability  to  collect  reliable  data  from  state,  local,  and  tribal  entities  is  a 
challenge  that  must  be  resolved  for  a  national  preparedness  performance  management 

10  GAO,  FEMA  Etas  Made  Progress,  20. 

11  Ibid.,  20-21. 

12  Department  of  Homeland  Security  Office  of  Inspector  General,  FEMA 's  Preparedness  for  the  Next 
Catastrophic  Disaster  —An  Update  (OIG-10-123)  (Washington,  DC:  DHS  September  2010),  55, 
https://www.oig.dhs.gov/assets/Mgmt/OIG_10-123_SeplO.pdf. 

13  Government  Accountability  Office  (GAO),  FEMA  Has  Made  Limited  Progress  in  Efforts  to 
Develop  and  Implement  a  System  to  Assess  National  Preparedness  Capabilities  (GAO-1 1-5 1R) 
(Washington,  DC:  Government  Accountability  Office,  2013),  19, 
http://www.gao.gov/assets/100/97160.pdf. 
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system  to  reach  fruition.  Stakeholders  adopting  a  congruent,  comprehensive  future 
assessment  system  are  vital  to  the  system’s  success.  While  there  is  no  data  indicating 
previously  developed  systems  were  difficult  to  use,  the  ease  of  use,  value,  and  reliability 
of  the  system’s  output  could  potentially  influence  future  federal  resource  allocation 
decisions.  This  concern  could  affect  the  perceived  relative  advantage  of  any  new 
comprehensive  assessment  system.  Complexity  of  the  system  will  affect  ease  of  use  and 
therefore  will  likely  be  a  significant  concern  to  the  stakeholders.  There  are  limited 
staffing  resources  at  the  operational  level,  and  any  system  requiring  considerable  effort 
and  staff  hours  to  meet  the  reporting  requirements  will  likely  meet  considerable 
resistance. 

This  challenge  was  identified  by  FEMA  in  2009  during  congressional  testimony. 
The  testimony  included  the  following:  “While  these  new  initiatives  have  bolstered  our 
nation’s  level  of  preparedness,  they  have  also  created  new  federal  requirements  for  state, 
territory,  local,  tribal  and  territorial  emergency  management  and  homeland  security 
agencies.  FEMA’s  key  partners  in  emergency  management  and  homeland  security  report 
that  the  existing  volume  of  requests  for  information  is  placing  a  significant  strain  on  their 
resources.”14 

Almost  from  its  inception,  pressure  arose  for  the  DHS  to  exhibit  the  value  and 
outcomes  attained  from  the  now  over  $40  billion  in  various  preparedness  grant 
programs.15  Due  to  the  increased  congressional  scrutiny  and  lack  of  perfonnance 
reporting,  these  grant  programs  have  seen  a  reduction  in  allocation  over  the  years. 
Resultantly,  Congress  limited  preparedness  grant  funding  in  the  2011  through  2013 
budget  cycles;  budget  requests  yielded  a  reduction  of  $875  million,  $1.28  billion,  and 


14  Preparedness:  What  Has  $29  Billion  in  Homeland  Security  Grants  Bought  and  How  Do  We  Know? 
Hearing  Before  the  House  Committee  on  Homeland  Security,  Subcommittee  on  Emergency 
Communications,  Preparedness  and  Response  9  (statement  of  Timothy  Manning,  Deputy  Administrator, 
Protection  and  National  Preparedness,  FEMA,  DHS),  1 1 1th  Cong.,  4,  (2009),  http- 
chsdemocrats.house.gov-SiteDocuments-20091027102505-12931,  accessed  May  18,  2013. 

15  Department  of  Homeland  Security  ,  “DHS  Announces  Grant  Allocations  for  Fiscal  Year  (FY)  2014 
Preparedness  Grants,”  July  25,  2014,  http://www.dhs.gov/news/2014/07/25/dhs-announces-grant- 
allocations-fiscal-year-fy-20 14-preparedness-grants. 
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$400  million  respectively. 16  That  FEMA  could  not  demonstrate  how  federal  grants 

enhanced  disaster  preparedness  was  a  contributing  factor  in  each  fiscal  year  reduction. 17 

In  testimony  before  the  House  Subcommittee  on  Emergency  Preparedness, 

Response  and  Communications,  Mark  Ghilarducci,  California  state  director  for 

Emergency  Services,  testified  to  the  impact  of  reduced  funding: 

While  the  number  of  threats  and  hazards  facing  states  and  the  nation  has 
increased,  federal  support  for  state  and  local  preparedness  efforts  has 
steadily  decreased.  Federal,  non-disaster  preparedness  grant  funding  has 
dropped  75  percent  since  2003.  This  reduction,  combined  with  state  and 
municipal  budget  challenges,  significantly  limited  the  ability  of  state  and 
local  governments  to  build  new  capabilities,  sustain  prior  investments,  and 
maintain  forward  momentum  with  preparedness  efforts.18 

The  reduction  in  spending  authority  for  SLTT  jurisdictions  since  2003  is 
significant  and  noteworthy. 

Analysis  of  the  DHS  budget  since  2008  reveals  an  interesting  trend  in  funding.  While 
the  overall  net  budget  for  DHS  increased  approximately  11  percent  over  the  seven-year 
period,  the  Homeland  Security  Grant  Program  (HSGP)  (which  currently  includes  three 
general  funding  categories:  the  State  Homeland  Security  Grant  Program  [SHSP],  the  Urban 
Area  Security  Initiative  [UASI],  and  Operation  Stonegarden  [OPSG])  decreased 
approximately  38  percent  overall.  Funding  to  SHSP  and  OPSG  endured  a  53  percent  funding 
reduction  while  the  UASI  cities  decreased  32  percent.19  In  general,  the  DHS  funding 
portfolio  has  experienced  a  steady  increase  since  it  was  founded,  while  the  distribution  to 
state  and  local  jurisdictions  has  trended  steadily  downward.20  Is  this  trend  a  result  of  the 


16  Mauer,  National  Preparedness,  9. 

17  Mauer,  National  Preparedness,  9,  Note  16. 

18  National  Governors  Association,  “Mark  Ghilarducci,  Testimony — A  State  Perspective  on  Assessing 
National  Preparedness  for  Disasters,”  September  19,  2013,  http://www.nga.org/cms/home/federal- 
relations/nga-testimony/hsps-testimony/co  12-content/ main-content-list/testimony— a-state-perspective- 
o.html. 

19  Department  of  Homeland  Security  (DHS),  Budget-in-Brief  Fiscal  Years  2008-2014  (Washington, 
DC:  DHS),  http://www.dhs.gov/dhs-budget,  accessed  October  4,  2014. 

20Preparedness  What  Has  $29  Billion  in  Homeland  Security >  Grants  Bought  and  How  Do  We  Know?, 
Hearing  Before  the  Subcommittee  on  Emergency  Communications,  Preparedness,  and  Response,  111th 
Cong,  (statement  of  Henry  Cuellar,  Chairman,  Subcommittee  on  Emergency  Communications, 
Preparedness  and  Response). 
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inability  of  SLTT  jurisdictions  to  report  on  their  preparedness  efforts,  a  deliberate  shift  in 
priorities  of  the  federal  government,  or  a  combination  of  the  two? 

A  review  of  three  seminal  documents,  the  DHS  Strategic  Plan,  the  Quadrennial 
Homeland  Security  Review  (QHSR)  and  current  preparedness  grant  guidance,  reveals  that 
it  is  important  for  the  whole  community  to  be  involved  in  national  preparedness.  The 
QHSR  identifies  the  whole  of  community  partnership  as  “individuals  and  families, 
including  those  with  access  and  functional  needs;  businesses;  faith-based  and  community 
organizations;  nonprofit  groups;  schools  and  academia;  media  outlets;  and  all  levels  of 
government,  including  federal,  state,  local,  tribal,  and  territorial  partners.”21  Preparedness 
grant  guidance  indicates  the  HSGP  “plays  an  important  role  in  the  implementation  of  the 
National  Preparedness  System  by  supporting  the  building,  sustainment,  and  delivery  of 
core  capabilities  essential  to  achieving  the  National  Preparedness  Goal  (the  Goal)  of  a 
secure  and  resilient  Nation.  The  building,  sustainment,  and  delivery  of  these  core 
capabilities  are  not  exclusive  to  any  single  level  of  government,  organization,  or 
community,  but  rather,  require  the  combined  effort  of  the  whole  community.”22 

The  latest  DHS  Strategic  Plan  and  QHSR  also  emphasize  the  importance  of 
partnerships  within  the  whole  community.  DHS’s  strategic  plan  identifies  preparedness 
grant  programs  as  a  central  activity  in  building  and  sustaining  national  core  capabilities,23 
while  the  QHSR  defines  three  core  principles  for  the  Whole  Community  approach: 

(1)  understanding  and  meeting  the  actual  needs  of  the  Whole  Community; 

(2)  engaging  and  empowering  all  parts  of  the  community;  and 

(3)  strengthening  what  works  well  in  communities  on  a  daily  basis.24 

21  Department  of  Homeland  Security  (DHS),  The  2014  Quadrennial  Homeland  Security  Review 
(Washington,  DC:  Department  ofHomeland  Security,  2014),  74, 
http://www.dhs.gov/sites/default/files/publications/qhsr/2014-QHSR.pdf. 

22  Department  ofHomeland  Security  (DHS),  Funding  Opportunity  Announcement  (FOA),  FY  2014, 
Homeland  Security  Grant  Program  (HSGP)  (Washington,  DC:  Department  ofHomeland  Security),  3, 
http://www.fema.gov/media-library/assets/documents/92397. 

23  Department  ofHomeland  Security  (DHS),  Fiscal  Years  2014-2018  Strategic  Plan  (Washington, 

DC:  Department  ofHomeland  Security),  36,  http://www.dhs.gov/sites/default/files/publications/FY14- 
18%20Strategic%20Plan.PDF. 

24  DHS,  The  2014  Quadrennial  Homeland  Security  Review,  73. 
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As  the  core  funding  for  these  efforts  continues  to  shift  toward  the  federal 
government,  is  it  most  effective  and  efficient  for  the  federal  government  to  assume  the 
primary  role  in  engaging  the  Whole  Community,  or  is  that  role  better  suited  to  regional, 
state,  or  local  governments? 

The  Heritage  Foundation  believes  ample  capability  should  already  exist,  given  the 
amount  of  resources  expended  to  date,  and  therefore  are  calling  for  Congress  to  limit 
federal  funding  toward  proven  and  demonstrated  needs.25  The  Foundation  argues  that 
low-risk  states,  cities,  and  infrastructure  entities  “have  received  more  than  enough  federal 
funds  to  meet  whatever  minimal  terrorism  threat  they  may  face.”26  Another  disapproving 
analysis  of  DHS  and  homeland  security  funding  comes  from  two  scholars,  Mueller  and 
Stewart,  who  argue  DHS  lacks  proper  justification  for  expenditures  based  on  threat 
probabilities  and  risk,  and  therefore  does  not  achieve  a  positive  return  on  investment.27 

The  phrase  “performance  management  system”  is  often  associated  with  human 
resource  departments  and  the  evaluation  and  management  of  people.  For  the  purpose  of 
this  research,  “performance  management  system”  is  used  in  a  generic  sense  to  encompass 
all  aspects  of  the  DHS  organizational  enterprise,  including  processes,  measurement, 
management,  culture,  planning,  and  its  people.  Many  process  improvement  programs 
have  gained  prominence  over  the  years  in  business  to  include  quality,  total  quality 
management,  six  sigma,  business  process  reengineering,  theory  of  constraints,  and  lean  to 
name  just  a  few.  All  of  these  programs  use  various  tools  and  techniques  that  provide  a 
common  approach  and  framework  to  improve  organizational  efficiencies  and 
effectiveness.  Perfonnance  management  systems  can  chose  to  implement  any  one  of 


25  Are  We  Prepared?.  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
Matt  A.  Mayer,  Visiting  Fellow,  Douglas  and  Sarah  Allison  Center  for  Foreign  Policy  Studies,  The 
Heritage  Foundation). 

26  Are  We  Prepared?,.  Measuring  the  Impact  of  Preparedness  Grants  Since  9/1 1,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
Matt  A.  Mayer,  Visiting  Fellow,  Douglas  and  Sarah  Allison  Center  for  Foreign  Policy  Studies,  The 
Heritage  Foundation). 

27  John  Mueller  and  Mark  G.  Stewart,  “Terror  Security  and  Money:  Balancing  the  Risks,  Benefits,  and 
Costs  of  Homeland  Security,”  Annual  Convention  of  the  Midwest  Political  Science  Association  Chicago, 
1L,  April  2011,  http://politicalscience.osu.edU/faculty/jmueller//MIDllTSM.PDF. 
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these  frameworks  or  a  hybrid  of  approaches  that  best  meets  their  needs  and  goals.  The 
aim  of  this  research  is  not  to  identify  the  specific  tools  and  techniques  that  should  be 
applied  to  the  homeland  security  enterprise  to  improve  performance.  Rather,  some 
performance  management  concepts  are  explored  to  show  the  potential  of  what  could  be 
accomplished  given  the  enterprise  devotes  greater  focus  and  attention  to  the  matter. 

C.  RESEARCH  QUESTIONS 

The  primary  research  question  is  this: 

•  What  would  a  performance  management  model  for  use  in  state,  local, 
tribal,  or  territorial  (SLTT)  jurisdictions  look  like? 

The  secondary  research  questions  are  as  follows: 

•  What  are  the  considerations  and  elements  required  for  implementation? 

•  What  would  be  the  challenges  and  implementation  issues  for  such  a 
model? 

During  its  12 -year  history,  a  number  of  efforts  for  DHS  to  establish  a  viable 
performance  measurement  system  have  come  and  gone,  leaving  the  organization  unable 
to  report  the  value  of  its  programs  to  Congress.  DHS’s  emergency  management 
performance  grant  and  homeland  security  grant  programs  are  two  critical  funding  sources 
to  states  and  urban  areas.  These  two  grant  programs  are  under  continuous  pressure  to 
show  value  for  the  investment,  and  each  fiscal  year  they  are  at  risk  for  reduced  funding 
allocations.  Because  all  states  and  urban  areas  rely  on  these  federal  funds  to  build  and 
sustain  capability  for  local  programs,  the  ability  to  document  and  report  outcomes  will 
advance  the  programs’ — and  the  funding’s — viability  to  congressional  appropriators.  If 
the  programs’  success  can  be  shown,  DHS  can  argue  for  sustained  funding  through  a 
difficult  fiscal  environment,  and  potentially  for  additional  or  special  funding  to  address 
an  identified  gap  in  critical  capability. 

Although  2013  was  the  first  year  DHS  utilized  both  the  State  Preparedness 
Reports  (SPR)  and  the  state  Threat  Hazard  Identification  and  Risk  Assessments  (THIRA) 
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to  compile  the  Presidential  Policy  Directive-8  mandated  National  Preparedness  Report ,28 
ample  reviews  and  critiques  of  the  current  perfonnance  management  system  already 
existed.  One  of  the  primary  findings  from  the  DHS  Office  of  the  Inspector  General  (OIG) 
was  that  many  individual  states  lacked  a  comprehensive  perfonnance  measurement 
system.29 

As  currently  developed,  the  data  provided  by  the  SPR  and  THIRA  offer  input  to 
the  National  Preparedness  Report  and  are  a  key  input  to  DHS’  Annual  Performance 
Report  Priority  Goal  3:  Ensure  Resilience  to  Disasters  by  Strengthening  Disaster 
Preparedness  and  Response  Capabilities.30  The  significant  drawbacks  to  the  current 
assessment  and  report  structures  are  the  subjective  nature  of  the  assessments  via 
qualitative  measures,  and  the  self-assessment  construct.  Refining  the  processes,  measures 
(questions),  and  reporting  tools  would  garner  statistically  significant  infonnation  and 
contribute  to  the  national  preparedness  capabilities  assessment. 

As  stated  previously,  one  of  the  greatest  challenges  with  current  preparedness 
measurements  is  the  lack  of  quantitative  measures  used  to  assess  state  and  urban  area 
programs’  progress.  Developing  metrics  that  meet  the  appropriate  data  standards  for 
specificity,  measurability,  reliability,  and  timeliness — and  that  provide  the  critical 
information  leaders  and  managers  need  to  set  priorities  and  direction — is  perhaps  the 
most  difficult  undertaking  in  the  performance  measurement  system.  It  may  be  too  broad  a 
scope  to  accomplish  in  the  given  time.  To  address  this  challenge,  a  sample  of  well- 
defined  measurements  should  be  developed  as  a  guide  for  future  researchers  or  program 
managers. 


28  FEMA,  National  Preparedness  Report  (Washington,  DC:  Department  of  Homeland  Security, 

2013),  2,  http://www.fema.gov/media-library-data/20130726-1916-25045-0015/npr2013_fmal.pdf. 

29  Department  of  Homeland  Security  Office  of  Inspector  General,  Semiannual  Report  to  the  Congress: 
Prepare,  Respond,  and  Recover,  October  1,  2012  through  March  31,  2013  (Washington,  DC:  Department 
ofHomeland  Security,  2013),  15,  17,  21. 

30  Department  ofHomeland  Security,  Annual  Performance  Report.  Fiscal  Years  2012-2014 
(Washington,  DC:  Department  ofHomeland  Security,  2013),  50. 
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D.  HYPOTHESIS 

A  standardized  performance  management  model  can  be  successfully  developed 
and  implemented  in  SLTT  jurisdictions  that  will  inform  executive  decision-making 
across  the  homeland  security  enterprise  and  improve  the  overall  effectiveness  of 
preparedness  activities  through  budget  allocations.  There  are  concerns,  however,  in 
realizing  and  sustaining  the  overall  effort.  The  first  concern  is  identifying  valid,  balanced, 
and  relevant  metrics  that  will  provide  reliable  and  actionable  data.  In  tandem  with  this 
challenge  is  establishing  the  resources  necessary  to  collect,  analyze,  and  report  the  data. 
Implementation  of  a  perfonnance  management  system  will  likely  require  an  initial 
investment  of  human  capital  as  well  as  establishing  training  programs  and  providing 
technical  assistance. 

E.  RESEARCH  METHOD  AND  SCOPE 

This  research  develops  a  conceptual  model  based  on  the  identified  shortfalls  from 
current  statutes  and  policies.  A  review  is  conducted  of  the  policy  and  guidance  directing 
federal  agencies  and  all  jurisdictions  receiving  federal  aid  to  track  and  report  perfonnance 
metrics.  In  addition,  prior  attempts  to  satisfy  the  reporting  requirements  are  analyzed.  To 
begin  the  process  of  developing  a  performance  management  model,  DHS  and  state 
policies,  guidance,  plans,  and  reports  are  reviewed  to  capture  goals  and  requirements. 
Existing  performance  management  models  for  organizations  that  conduct  similar 
missions  are  evaluated  for  strengths  and  weaknesses  as  they  apply  to  the  homeland 
security  organization. 

Through  this  analysis,  along  with  the  application  of  process  improvement  theory 
and  techniques,  a  generic  data  collection  model  shows  the  alignment  of  SLTT  strategic 
goals  and  objectives  and  performance  measures  to  federal  goals  and  objectives.  The 
model  accounts  for  the  unique  aspects  of  individual  jurisdictions;  the  priorities  and  focus 
areas  of  each  are  based  on  threats,  capabilities,  and  vulnerabilities  requiring  distinctive 
measurements  to  capture  the  required  data. 

The  goal  of  this  research  is  to  devise  a  perfonnance  management  model  that  will 
serve  SLTT  jurisdictions  in  providing  performance  management  infonnation.  In  addition, 
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it  endeavors  to  be  flexible  enough  to  address  individual  jurisdiction  priorities  while  at  the 
same  time  maintaining  sufficient  standardization,  allowing  the  collected  data  to  inform 
federal  departments  and  agencies  on  progress  and  utilization  of  funds.  This  research 
effort  serves  as  a  starting  point  for  the  performance  management  system.  Due  to  the 
complexity  of  the  DHS  mission  and  its  many  components,  further  research,  analysis,  and 
policy  development  will  be  required. 

F.  SIGNIFICANCE  OF  RESEARCH 

There  are  three  underlying  reasons  for  researching  a  construct  to  measure  the 
performance  of  homeland  security  programs.  Foremost  is  the  gain  in  efficiencies  and 
effectiveness  in  SLTT  homeland  security  programs  through  the  collection  and  analysis  of 
improved  data  and  metrics.  Efficiency  and  effectiveness  undergird  actual  system 
performance  and  serve  as  the  basis  for  measurement.  The  research  will  also  provide  a 
means  for  program  directors  to  evaluate  programmatic  outcomes  and  measure  progress 
toward  strategic  objectives.  Additionally,  the  use  of  data  can  promote  a  more  consistent 
application  of  policy  and  enhance  transparency  in  resource  allocation  decisions.  An 
established  perfonnance  measurement  framework  will  facilitate  communication  with  all 
of  the  stakeholders,  including  state  legislators  and  administrations,  headquarters  DHS, 
and  Congress. 

There  is  increased  recent  scrutiny  over  how  the  SLTT  jurisdictions  expend  their 
allocated  funds.31  As  previously  stated,  Congress  has  reduced  the  amount  of  federal  grant 
dollars  to  SLTT  programs  due,  in  part,  to  DHS’s  inability  to  report  on  the  value  created 
by  the  expenditure  of  funds.  Congress  may  well  decide  to  shift  the  fiscal  burden  of 
preparedness  programs  to  state  and  local  programs;  however,  that  decision  should  not 
come  as  a  result  of  an  inability  to  report  progress  on  established  goals.  A  standardized 
approach  to  SLTT  performance  measurement  could  provide  DHS  a  comparative  analysis 
and  reporting  tool  to  inform  Congress. 


3 1  Preparedness  What  Has  $29  Billion  in  Homeland  Security  Grants  Bought  and  How  Do  We  Know?, 
Hearing  Before  the  Subcommittee  on  Emergency  Communications,  Preparedness,  and  Response,  111th 
Cong,  (statement  of  Henry  Cuellar,  Chairman,  Subcommittee  on  Emergency  Communications, 
Preparedness  and  Response). 
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A  performance  management  system  can  also  be  used  by  both  state  and  federal 
program  executives  to  justify  program  expenditures  and  inform  critical  resource 
allocation  decisions  to  mitigate  risks  and  fill  resource  gaps.  Clear  perfonnance  data  will 
promote  consistent  application  of  policy  and  enhance  transparency  in  resource  allocation 
decisions.  Moreover,  an  established  performance  measurement  framework  will  facilitate 
communication  with  all  stakeholders,  including  state  legislators  and  administrations, 
headquarters  DHS,  and  Congress.  SLTT  entities’  inability  to  track  perfonnance  indicators 
is  a  systemic  problem.  As  the  DHS  Office  of  Inspector  General  noted  in  its  2013  semi¬ 
annual  report,  “Improvements  are  needed  in  the  following  areas:  developing  a 
comprehensive  strategy  with  measurable  objectives,  developing  a  performance 
measurement  system  to  assess  emergency  preparedness,  and  improve  the  perfonnance 
measurement  process.”32 

There  are  many  potential  benefits  to  incorporating  a  perfonnance  measurement 
framework  in  SLTT  homeland  security  programs.  Data  derived  from  the  framework  will 
facilitate  decision-making  at  all  levels  of  management  and  will  provide  timely  feedback 
on  the  efficacy  of  those  decisions.  Data  also  enables  accountability;  managers  will  gain 
insight  into  program  processes  and  organizational  compliance  with  policy,  guidance,  and 
directives.  Metrics  also  serve  as  a  tool  to  influence  behavior  and  place  direct  attention  on 
organizational  priorities. 

In  spite  of  all  the  challenges,  FEMA  states  they  can  measure  SLTT  preparedness 
capability  improvements  to  understand  how  the  jurisdictions  are  meeting  the  National 
Preparedness  Goal.  FEMA  asserts  this  is  accomplished  by  comparing  jurisdiction’s  self- 
identified  capability  gaps,  via  their  annual  Threat  Hazard  Identification  and  Risk 
Assessment  and  State  Preparedness  Reports,  with  their  planned  investment  strategy 
outlined  in  the  annual  preparedness  grant  application  package.33 


32  Department  of  Homeland  Security  Office  of  the  Inspector  General,  Semi-Annual  Report  to  the 
Congress. 

33  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/1 1,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
statement  of  Timothy  Manning,  Deputy  Administrator,  FEMA  for  Protection  and  National  Preparedness). 
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In  testimony  before  the  Senate  Emergency  Management  Subcommittee,  FEMA 
provided  the  following  appraisal  of  the  nation’s  current  preparedness: 

We  are  more  secure  and  better  prepared  than  we  have  been  at  any  time  in 
our  history;  we  plan  better,  organize  better,  equip  better,  train  better,  and 
exercise  better,  resulting  in  improved  national  preparedness  and  resilience; 
we  have  built  and  enhanced  capabilities  by  acquiring  needed  equipment, 
funding  training  opportunities,  developing  preparedness  and  response 
plans,  exercising  and  building  relationships  across  jurisdictional  lines.34 

While  all  of  these  claims  may  be  true,  there  is  no  understanding  of  how  much  better 
prepared  the  nation  is,  how  much  capability  is  required,  and  how  much  capability  was 
developed  over  the  past  ten  years.  Although  the  capability  to  respond  to  a  local  disaster 
involves  much  more  than  the  possession  of  assets,  an  audit  of  the  assets  acquired  through 
federal  grants  has  never  been  accomplished;35  therefore,  an  accurate  evaluation  of 
acquired  capability  can  never  be  accomplished.  Since  2009,  the  U.S.  Government 
Accountability  Office  (GAO)  has  recommended  DHS  develop  “clear,  objective,  and 
quantifiable  capability  requirements  and  performance  measures.”36  DHS  has  initiated 
many  attempts  to  establish  an  assessment  system  and  implemented  new  reporting 
requirements  to  meet  the  goal  of  assessing  national  preparedness,  but  has  yet  to  quantify 
the  assessment  through  the  development  of  perfonnance  measures.  This  inquiry  explores 
and  analyzes  the  various  systems  and  processes  put  in  place  by  DHS  and  attempts  to 
detennine  the  root  causes  for  failure,  in  order  to  detennine  system  perfonnance 
adequately. 

G.  CHAPTER  ORGANIZATION 

Chapter  II  provides  a  review  of  the  related  literature  and  focuses  on  three  main 
topics:  the  mandated  requirements  and  historical  efforts  of  DHS  to  implement  a 


34  Ibid. 

35  Are  We  Prepared?.  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security’  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
Matt  A.  Mayer,  Visiting  Fellow,  Douglas  and  Sarah  Allison  Center  for  Foreign  Policy  Studies,  The 
Heritage  Foundation.) 

36  Maurer,  National  Preparedness,  8. 
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performance  management  system,  the  potential  value  a  system  can  provide  an 
organization,  and  the  critical  elements  to  consider  in  developing  such  a  system.  Chapter 
III  takes  a  detailed  look  at  the  problem  space  and  analyzes  the  laws  and  directives  that 
dictate  perfonnance  management  requirements.  It  also  reviews  DHS’  attempts  to 
implement  performance  management  in  their  programs  and  identify  the  specific 
challenges  that  hinder  its  development.  Finally,  Chapter  IV  provides  potential  solutions 
to  the  identified  challenges  by  developing  a  conceptual  model  that  considers  policy,  roles 
and  responsibilities,  training,  staffing,  and  reporting. 
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II.  LITERATURE  REVIEW 


This  review  identifies  relevant  sources  concerning  the  requirement  for  and  value 
of  a  performance  measurement  system  within  DHS.  The  sources  fall  into  the  following 
categories:  the  requirement  for  a  performance  management  system;  the  value  of  a 
performance  management  system,  a  history  of  DHS  initiatives  to  meet  the  requirement; 
and  the  elements  of  a  successful  performance  management  system. 

Overall,  there  are  numerous  resources  from  congressional  testimony,  Government 
Accountability  Office  (GAO)  reports,  and  other  think  tanks  describing  the  requirement 
and  need  for  a  perfonnance  management  system.  There  are  also  many  resources 
providing  guidance  and  theory  in  performance  measurement  systems,  performance-based 
decision-making,  metrics  and  measurement,  and  strategic  planning.  Finally,  sources 
provide  a  reasonable  account  of  the  systems  implemented  by  DHS  over  the  past  ten 
years,  and  their  shortcomings.  Detailed  infonnation  regarding  system  capabilities  and 
user  interface,  however,  was  not  readily  available. 

A.  THE  REQUIREMENT  FOR  A  PERFORMANCE  MANAGEMENT 

SYSTEM 

Since  the  publication  of  HSPD-8  in  2003,  public  law  has  required  government  to 
improve  efficiency  and  effectiveness  of  their  programs  and  operations.  DHS  has  included 
these  goals  into  their  policy  directives  and  struggled  to  implement  and  achieve  them.  The 
GAO  has  provided  substantial  documentation  and  reporting  on  these  DHS  deficiencies 
through  reports  (09-369,  09-651)  and  congressional  briefs  (11-5 1R)  and  testimony  (13- 
637T).  These  documents  are  more  fully  examined  in  Chapter  III.  The  DHS  Office  of  the 
Inspector  General  (OIG)  has  also  reported  on  and  highlighted  DHS’s  difficulty  to  develop 
a  performance  management  system.  States  and  designated  urban  areas  have  received  over 
$40  billion  in  DHS  grant  funding  to  develop  preparedness  and  response  capabilities.  To 
better  account  for  this  spending,  Public  Law  110-53, 37  which  took  effect  in  2007, 


37  Implementing  Recommendations  of  the  9/1 1  Commission  Act  of 2007,  Pub.  L.  No.  110-53,  121  Stat. 
288-289  (2007). 
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directed  an  audit  of  every  state’s  federal  preparedness  grant  funding  program.  The  DHS 
OIG  provided  testimony  before  Congress  in  June  2013  that  highlighted  the  lack  of 
strategic  planning  and  measurable  goals  by  a  significant  number  of  states.38  Although  the 
DHS  OIG  did  not  audit  all  programs,  the  sample  of  audits  suggests  states  generally  do  not 
have  an  adequate  system  in  place  to  measure  and  report  on  preparedness  perfonnance 
indicators. 

Think  tanks  such  as  The  Cato  Institute  and  RAND  Corporation  have  also  weighed 
in  on  DHS’s  challenges  to  assess  national  preparedness.  While  RAND  acknowledges  the 
need  for  an  assessment  system,  it  also  highlights  the  limits  of  measuring  inputs  to  the 
disaster  response  system  as  a  means  of  indicating  predictable  outcomes.39  RAND  points 
out  that  it  is  difficult  to  assess  human  response  systems  with  processes  that  are  impossible 
to  control  and  measure  in  a  standardized  way.40  In  an  opposing  view,  The  Cato  Institute 
argues  that  more  than  enough  money  has  already  been  obligated  to  build  capability  in 
some  homeland  security  endeavors,  and  it  is  now  time  to  prioritize  federal  resources  to 
the  most  critical  programs  and  cut  funding  to  those  programs  that  should,  by  now,  be 
mature.41 

B.  THE  VALUE  OF  A  PERFORMANCE  MANAGEMENT  SYSTEM 

While  the  Government  Perfonnance  Results  and  Modernization  Act  (GPRA-MA) 
requires  an  assessment  and  annual  perfonnance  report  from  most  federal  agencies,  DHS’s 
performance  is  especially  important.  Director  of  the  RAND  Homeland  and  Security 
Defense  Center  Henry  Willis,  in  his  congressional  testimony,  outlines  three  aspects  of  the 


38  Are  We  Prepared?,  (statement  of  Anne  L.  Richards,  Assistant  Inspector  General  for  Audits,  DHS). 

39  Brian  A.  Jackson,  The  Problem  of  Measuring  Emergency  Preparedness:  The  Need  for  Assessing 
Response  Reliability  as  Part  of  Homeland  Security  Planning  (RAND  Document  No.  OP-234-RC)  (Santa 
Monica,  CA:  RAND,  2008),  11, 

http://www.rand.org/content/dam/rand/pubs/occasional_papers/2008/RAND_OP234.pdf. 

40  Brian  A.  Jackson,  The  Problem  of  Measuring  Emergency  Preparedness:  The  Need  for  Assessing 
Response  Reliability  as  Part  of  Homeland  Security  Planning  (RAND  Document  No.  OP-234-RC)  (Santa 
Monica,  CA:  RAND,  2008),  11, 

http://www.rand.org/content/dam/rand/pubs/occasional_papers/2008/RAND_OP234.pdf. 

41  David  Rittgers,  Abolish  the  Department  ofHomeland  Security  (Policy  Analysis  #683), 
(Washington,  DC:  Cato  Institute,  2011),  8-9,  www.cato.org/publications/policy-analysis/abolish- 
department-homeland-security-20 1 1 -09-08 . 
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DHS  mission  that  are  critical  to  national  security:  1)  deciphering  poorly  understood 
threats,  2)  mitigating  the  national  consequences  of  these  threats,  and  3)  developing  an 
integrated  approach  to  navigate  the  complexity  of  many  DHS  and  external  agencies.42 

Given  the  mandate  to  develop  and  put  into  operation  a  perfonnance  management 
system,  the  logical  question  that  follows  is:  What  will  the  agency  gain  from 
implementation?  Resource  constrained  executives  and  managers  will  struggle  to  invest 
resources  to  a  system  when  they  do  not  know  if  return  in  savings  or  additional  capability 
are  worth  the  investment;  “an  expensive  performance  management  system,  even  if  it 
accomplishes  something,  may  not  accomplish  enough  to  justify  the  expense.”43 

Independent  research  in  three  separate  studies  by  Hendricks  and  Singhal;  Powell, 
Eaton  and  Jarrell;  and  Leuschner  all  found  evidence  supporting  improvements  in 
economic  and  operational  perfonnance  from  the  implementation  of  perfonnance 
management  systems  in  both  public  and  private  organizations.44  Eaton  and  Jarrell’s 
research  indicated  organizations  with  a  highly  developed  or  advanced  system 
implementation  attained  even  stronger  perfonnance  levels.45  Hendricks  and  Singhal 
suggest  that  organizations  implementing  performance  management  systems  at  a  level  to 
receive  award  recognition  also  outperfonn  their  competition.46  This  is  supported  by  the 


42  Henry  H.  Willis,  Strengthening  Strategic  Planning  and  Management  at  DHS  (RAND  Testimony 
No.  CT-387)  (Santa  Monica,  CA:  RAND,  2013),  4, 

http://www.rand.org/content/dam/rand/pubs/testimonies/CT300/CT387/RAND_CT387.pdf. 

43  James  J.  Heckman,  Carolyn  Heinrich,  and  Jeffrey  Smith,  The  Performance  of  Performance 
Standards  (Working  Paper  9002)  (Cambridge,  MA:  The  National  Bureau  ofEconomic  Research,  2002), 11, 
http://www.nber.org/papers/w9002. 

44  Kevin  B.  Hendricks  and  Vinod  R.  Singhal,  “Does  Implementing  an  Effective  TQM  Program 
Actually  Improve  Operating  Performance?  Empirical  Evidence  from  Firms  That  Have  Won  Quality 
Awards,”  Management  Science,  43  (September  1997),  1271,  http://www.jstor.org/stable/2634637;  Thomas 
C.  Powell,  “Total  Quality  Management  as  Competitive  Advantage:  A  Review  and  Empirical  Study,” 
Strategic  Management  Journal,  Vol.  16,  No.  1  (Jan  1995),  29,  http://links.jstor.org/sici7sicH0143- 
2095%28 19950 1%2916%3A1%3C15%3ATQMACA% 

3E2.0.CO%3B2-5;  George  S.  Eaton  and  Sherry  L.  Jarrell,  “The  Effects  of  Total  Quality  Management  on 
Corporate  Performance:  An  Empirical  Investigation,”  The  Journal  ofBusiness,  Vol.  71,  No.  2  (April  1998), 
298,  http://www.jstor.org/stable/10.1086/209744;  Kristin  J.  Leuschner,  Are  Performance-based 
Accountability  Systems  Effective?:  Evidence  From  Five  Sectors  (RAND  Research  Brief  No.  RB-9549) 
(Santa  Monica,  CA:  RAND  Education,  2010),  2, 

http://www.rand.org/content/dam/rand/pubs/research_briefs/2010/RAND_RB9549.pdf. 

45  Eaton  and  Jarrell,  “The  Effects  of  Total  Quality  Management,”  298. 

46  Hendricks  and  Singhal,  “Does  Implementing  an  Effective  TQM  Program  Actually  Improve 
Operating  Performance?,”  1271. 
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Bureaus  of  Economic  Analysis  and  Labor  Statistics,  which  shows  organizations  earning 
two  Baldrige  Awards  for  perfonnance  excellence  demonstrate  median  growth  in  jobs 
nearly  20  times  greater  than  matched  industries  and  time  periods.47  In  addition,  these 
organizations  have  seen  a  92.6  percent  increase  in  median  revenue  growth  and  a  67 
percent  median  growth  in  number  of  operational  locations.48 

Eaton  and  Jarrell  correlated  long-term  organizational  benefits  to  an  advanced  state 
of  performance  improvement  programs,  suggesting  “it  takes  time  for  an  organization  to 
reap  the  benefits  of  implementation.”49  Hendricks  and  Singhal’s  research  also  indicates 
there  are  no  negative  performance  indicators  during  the  early  implementation  phase, 
despite  the  costs  of  program  implementation,  due  to  the  offset  of  short-term  benefits.50 

While  research  suggests  organizations  can  benefit  from  implementing 
performance  improvement  programs,  findings  also  illuminate  the  challenges  to  successful 
implementation.  Powell’s  research  on  the  value  of  perfonnance  improvement  programs 
found  the  results  are  not  consistent  for  all  implementers,51  and  it  is  more  challenging  for 
large  organizations  to  achieve  performance  improvement.52  Leuschner’s  study  suggests 
that  systems’  specific  design  elements  are  critical  to  implementing  the  program 
successfully.53  These  design  elements  are  examined  more  thoroughly  in  the  next  section. 

C.  ESSENTIAL  ELEMENTS  OF  A  SUCCESSFUL  PERFORMANCE 

MANAGEMENT  SYSTEM 

A  review  of  the  literature  highlighted  six  common,  overarching  elements  critical 
to  successfully  implementing  a  perfonnance  management  system.  The  six  themes  are: 
management,  measurement,  planning,  communication,  incentives,  and  employee 

47  Baldrige  Performance  Excellence  Program,  “The  Baldrige  Criteria  101,” 
http://www.nist.gov/baldrige/publications/upload/The-Baldrige-Criteria-101.docx,  Accessed  June  6,  2015. 

48  “The  Baldrige  Criteria  101.” 

49  Eaton  and  Jarrell,  “The  Effects  of  Total  Quality  Management,”  298. 

50  Hendricks  and  Singhal,  “Does  Implementing  an  Effective  TQM  Program  Actually  Improve 
Operating  Performance?,”  1271. 

51  Powell,  “Total  Quality  Management,”  29. 

52  Ibid.,  26. 

53  Leuschner,  Are  Performance-based  Accountability  Systems  Effective?,  2. 
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involvement.  These  elements  are  addressed  individually  in  this  section,  but  most  of  them 
are  interrelated,  if  not  mutually  supporting. 


1.  Management 

Of  the  six  themes,  management  is  perhaps  the  most  critical.  Organizational 
leaders  have  much  influence  within  the  remaining  five  elements.  They  directly  impact 
priorities,  and  play  a  key  role  in  holding  organizational  departments  accountable  to  the 
improvement  process.  The  independent  research  of  Powell;  Rosett  and  Rosett;  and 
Bourne  Neely,  Platts,  and  Mills  all  found  management’s  role  to  be  a  central  factor  in  the 
success  of  performance  improvement  initiatives.54 

Light’s  research  identified  poor  management  execution  as  a  primary  root  cause  of 
government  failures,55  while  Moynihan  and  Lavertu  indicate  management’s  lack  of 
involvement  and  use  of  perfonnance  information  as  significant  contributors.56  Eaton  and 
Jarrell  provided  a  strong  correlation  between  management  methods  and  organizational 
performance  improvement,57  while  Powell’s  research  states  executive  commitment  is 
more  important  to  success  than  the  various  tools  and  techniques  of  an  improvement 
program,  such  as  benchmarking,  training,  and  improved  measurement.58  (Tools  and 
techniques  will  be  described  in  more  detail  in  Chapter  IV.) 


54  Powell,  “Total  Quality  Management  as  Competitive  Advantage,  29;  James  G.  Rosett  and  Richard 
N.  Rosett,  Characteristic  of  TQM:  Evidence  From  the  RIT/CS4  Today  Quality  Cup  Competition  (Working 
Paper  7241)  (Cambridge,  MA:  The  National  Bureau  ofEconomic  Research,  1999), 14, 
http://www.nber.org/papers/w7241;  Mike  Bourne  et  at,  “The  Success  and  Failure  of  Performance 
Measurement  Initiatives:  Perceptions  of  Participating  Managers,”  International  Journal  of  Operations  & 
Production  Management  22  (2002):  1303,  doi:  10.1108/01443570210450329. 

55  Paul  C.  Light,  A  Cascade  of  Failures:  Why  Government  Fails  and  How  to  Stop  It,  (Washington, 
DC:  Brookings  Institute,  2014),  8,  http://www.brookings.edu/research/papers/2014/07/14-cascade-failures- 
why-government-fails-light. 

56  Donald  Moynihan  and  Stephane  Lavertu,  Do  Performance  Reforms  Change  How  Federal 
Managers  Manage  (Washington,  DC:  Brookings  Institute,  2014),  8,  http://www.brookings.edu/research/ 
papers/20 12/10/1 1  -management-mo  ynihan. 

57  Eaton  and  Jarrell,  “The  Effects  of  Total  Quality  Management  on  Corporate  Performance,”  298. 

58  Powell,  “Total  Quality  Management  as  Competitive  Advantage,”  29. 


21 


2. 


Measurement 


Metrics  development  is  a  critical  component  in  assessing  performance,  and  is 
perhaps  one  of  the  most  challenging  elements  to  implement  effectively.  Pyzdek  describes 
measurement  as  “the  assignment  of  numbers  to  observed  phenomena  according  to  certain 
rules.”59  These  “rules”  are  important  to  ensure  the  numbers  have  relevancy  and  meaning, 
which  will  help  managers  conduct  analysis  and  make  decisions.  Pyzdek  identifies  two 
fundamental  properties  that  guide  the  collection  of  metrics:  validity  and  reliability.  Data 
validity  ensures  the  “item  measures  what  is  intended”  and  reliability  “ensures  a 
measurement  orders  individual  responses  in  the  same  way.”60 

Asch  et  al.  describe  the  ideal  measurement  system  as  one  containing  elements 
from  three  categories:  structure,  process,  and  outcomes  (though  outcome  measures  in  the 
area  of  preparedness  are  rare  and  difficult  to  ascertain).61  Structural  measures  have  the 
advantage  of  being  most  responsive  to  policy  changes  but  perhaps  least  related  to 
outcomes.62  Process  measures  are  most  responsive  to  quality  improvement  efforts  and 
are  more  directly  related  to  outcomes.63  Heckman,  Heinrich,  and  Smith  describe  a  fourth 
category  of  measurement — impact.  Perfonnance  measures  based  on  impacts  are  likely  to 
be  more  controversial,  depending  on  the  data  collection  method,  and  are  more  costly  to 
produce  than  outcome  measures  due  to  the  additional  time  and  analysis  involved.64 

The  GAO  identified  the  “long-standing  challenge  of  measuring  and  collecting 
accurate  performance  data”65  as  the  greatest  shortcoming  agencies  face  in  assessing  and 
reporting  their  GPRA-MA-mandated  performance  progress.  Bourne  et  al.  identified  a  set 

59  Thomas  Pyzdek,  The  Six  Sigma  Handbook,  A  Complete  Guide  for  Green  Belts,  Black  Belts,  and 
Managers  at  All  Levels  (New  York:  McGraw-Hill,  2003),  277. 

60  Ibid.,  280. 

61  Stephen  M.  Asch  et  at,  “A  Review  of  Instruments  Assessing  Public  Health  Preparedness,”  Public 
Health  Reports,  Vol.  120  (September-October  2005),  535,  http://www.ncbi.nlm.nih.gov/pmc/articles/ 

PMC  1497752/pdf  16224986.pdf,  accessed  April  7,  2015. 

62  Asch  et  at,  “A  Review  of  Instruments  Assessing  Public  Health  Preparedness,”  535. 

63  Ibid.,  539. 

64  Heckman,  Heinrich,  and  Smith,  The  Performance  of  Performance  Standards,  1 1 . 

65  Government  Accountability  Office  (GAO),  Managing  for  Results:  Agencies  Should  More  Fully 
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of  key  challenges  organizations  face  in  developing  a  measurement  system:  poorly  defined 
metrics,  the  lack  of  a  highly  developed  information  system,  too  many  measures  diluting 
the  overall  impact,  and  the  need  to  collect  more  quantifiable  versus  qualitative  results.66 
Additionally,  Leuschner  asserts  that  in  order  to  collect  performance  data  successfully,  the 
associated  metrics  must  be  unambiguous  and  easy  to  observe.67 

Another  challenge  with  developing  actionable  measures  is  the  time  lag  between 
the  observed  measure  and  subsequent  analysis,  and  a  manager  or  leader’s  need  to  make 
short-term  decisions.68  Therefore,  “if  performance  standards  are  to  be  put  in  place  that 
motivate  efficiency  in  the  short-tenn,  the  perfonnance  management  system  needs  to 
detennine  which  short  run  measures  are  strongly  related  to  long  tenn  efficiency 
criteria.”69  The  conflict  of  collecting  short-term  management  measures  for  expedient 
decision-making,  while  also  trying  to  measure  long-term  progress  of  strategic  goals  is 
evident  in  DHS.  Smith’s  research  reveals  a  disconnect  between  the  GPRA-MA  mandate 
and  DHS’s  solution,  resulting  in  a  bi-furcated  measurement  practice.  DHS  collects  one 
set  of  short-tern,  politically  driven  measures  to  satisfy  GPRA-MA  requirements,  and  a 
second  set  aligned  with  its  own  internal  agency  measures.70 

A  good  measure  should  include  clear  standards,  with  the  required  data  elements 
explicitly  detailed.71  Davis  et  al.  found  that  standardized  measures  are  essential  for 
assessing  state  and  local  organizations’  preparedness  levels.72  Applying  these  standards 
to  SLTT  jurisdictions  receiving  federal  grant  program  dollars  can  encourage  and  produce 
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greater  accountability  in  the  use  of  those  funds.73  Without  measurement  standards, 
organizations  attempt  to  fill  duplicate  information  requests,  which  results  in  wasteful 
activity.74  In  addition,  inadequate  standards  can  lead  to  poor  investments  that  fail  to 
improve  outcomes  or  build  capabilities  to  meet  current  and  future  needs.75  While 
preparedness  standards  are  considered  the  backbone  of  accountability — which  should 
lead  to  institutionalized  and  sustained  preparedness — Canada’s  research  also  warned  that 
rigid  standards  can  inhibit  SLTT  jurisdictions  ability  to  find  approaches  that  best  meet  the 
needs  of  each  community  based  on  unique  geography,  demographics,  organizational 
structure  and  resources.76 

The  abundance  of  available  SLTT  performance  measures  collected  through  State 
Preparedness  Reports  are  predisposed  to  checklist  items,  yes  or  no  questions,  or  Likert 
Scale  assessment  questions  that  use  a  subjective,  self-reported  three-  or  five-tier  ranking 
scale.  While  there  is  value  in  collecting  nominal  or  attribute  data  through  these  survey 
instruments,  these  data  are  considered  the  weakest  form  of  measurement.77 

Jackson’s  research  shows  that  self-collection  is  a  common  means  of  data 
collection,  and  may  be  necessary  to  gain  expert  opinions.78  According  to  an  American 
Journal  of  Public  Health  article,  however,  this  technique  yields  biased  outcomes  with 
“ambiguous  and  uncertain  preparedness  goals,  a  lack  of  agreement  about  what  the 
measures  should  aim  at  and  how  they  should  be  interpreted,  and  a  weak  system  of 
accountability  for  producing  results.”79  The  article  goes  on  to  state,  “Measures  often  vary 
considerably  across  agencies  and  shift  dramatically  from  year  to  year,  leaving  state  and 
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local  health  officials,  businesses,  nonprofits,  and  citizens  confused  and  perplexed  by  a 
maze  of  overlapping  and  sometimes  contradictory  requirements,  checklists,  and  ideas 
about  what  constitutes  preparedness.”80 

3.  Strategic  Planning 

The  strategic  planning  process  requires  a  focused  and  diligent  effort  to  identify 
priorities  and  develop  action  plans  that  shape  the  nature  and  direction  of  an  entity’s 
activities.  But  perhaps  the  most  concise  definition  of  strategy  is  “a  plan  for  success.”81 
Bryson  concludes  that  “strategic  thought  and  action  [emphasis  added]  are  increasingly 
important  to  the  continued  viability  and  effectiveness  of  governments  and  is  purposed  to 
help  organizations  respond  effectively  to  new  situations.”82 

Organizational  strategy  is  formed  as  a  result  of  a  deliberate  planning  process,  and 
the  resulting  strategy  defines  how  the  organization  intends  to  achieve  its  established 
vision.  The  strategy  determines  the  best  way  to  use  available  resources  to  achieve  the 
planned  goals  or  outcomes,83  and  should  contain  few  competing  interests  or 
requirements.84  DHS  OIG  frequently  identifies  strategic  planning  as  a  systemic 
deficiency  throughout  the  SLTT  jurisdictions;  it  has  found  many  states  lack  an  existing  or 
current  strategic  plan.  But,  more  importantly,  most  plans  it  did  find  lacked  sufficient 
goals  and  objectives  to  drive  management  decision-making.85 

Bourne  et  al.  identified  several  common  shortfalls  of  strategic  planning  efforts 
that  contribute  to  the  failure  of  performance  improvement  initiatives: 
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•  Lack  of  leadership  and  resistance  to  change 

•  Unwillingness  to  commit  required  time  and  resources 

•  Vision  and  strategy  were  not  actionable 

•  Strategy  was  not  linked  to  resource  allocation 

•  Strategy  was  not  linked  to  department,  team,  and  individual  goals 
(alignment) 

•  Goals  were  negotiated  rather  than  based  on  stakeholder  requirements 

•  Strategic  plans  are  central  to  all  organizational  activities  and  involve  all 
employees.  The  shortfalls  identified  by  Bourne  et  al.  are  traits  that 
contribute  to  the  onset  of  the  strategic  plan’s  failure.  If  mid-level  managers 
and  staff  fail  to  see  a  connection  between  their  individual  or  departmental 
efforts  and  the  organizational  strategy,  there  will  be  no  incentive  to  meet 
the  identified  goals  and  objectives  within  the  strategy.86 

4.  Incentives 

Staff  motivation  is  important  if  management  wishes  to  incorporate  a  perfonnance 
management  system.  Perfonnance  measures  need  to  provide  value  to  organizational  staff 
as  well  as  the  program  managers,  either  through  process  value  or  through  an  incentive 
mechanism  that  works.  Performance  measures  can  motivate  improved  performance  by 
simplifying  processes  and  procedures  of  routine  staff  functions,  or  through  a  system  of 
linked  rewards,  benefits,  or  penalties.87 

Incentives  should  apply  to  individuals  who  or  organizations  that  have  control  over 
the  relevant  inputs  and  processes,  and  should  be  meaningful  to  those  being 
incentivized.88  The  practice  of  tying  rewards  to  accurately  measurable  perfonnance  or 
management,  however,  is  rare.89  Metzenbaum’s  research  shows  there  is  a  correlation 
between  the  size  of  rewards  or  incentives  and  the  staffs  willingness  to  accept  or  embrace 
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the  system.90  Poorly  designed  incentives  can  give  rise  to  unanticipated  and  undesired 
consequences.  Too  often,  managers  rely  on  the  best  available  or  easiest  measure  to 
collect  instead  of  the  best  measure  to  determine  performance.91  This  can  result  in  faulty 
analysis  and  ill-advised  incentives. 

5.  Communication 

Powell’s  research  found  an  “open  organization”  is  critical  to  determine 
performance  improvement  initiatives’  successes  or  failures.92  Openness  requires  a  free- 
flowing  dialogue  between  management,  staff,  and  stakeholders.  Light’s  research  shows 
that  organizational  missions  that  are  not  communicated  and  embraced  can  be  easily 
undermined  by  rank  corruption  and  unethical  conduct;  clearly  defined  missions  that  are 
embraced  across  the  organization,  however,  can  be  effectively  monitored  and  corrected 
through  performance  measurement  and  management.93 

Other  research  indicates  organizational  goals  should  be  widely  shared  among  all 
stakeholders.94  Strategic  communications  can  reinforce  the  organization’s  vision  and 
create  an  environment  in  which  employees  are  guided  by  the  firm’s  broader  objectives. 
Additionally,  a  well-articulated  vision  that  includes  a  focus  on  customer  satisfaction  is  a 
key  element  of  a  good  communication  strategy.95 

6.  Culture 

Organizational  culture  plays  a  strong  role  in  the  successful  implementation  and 
execution  of  a  perfonnance  management  system.  Research  by  Vest  and  Gamin  highlights 
two  essential  elements:  a  wholesale  change  in  business  practices  and  a  fundamental 
change  in  culture.  “The  inability  of  many  organizations  to  ensure  transformation  along 
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both  these  dimension  [sic]  may  explain  a  number  of  previous  failings  of  lauded 
[performance  improvement]  approaches  to  be  viewed  by  employees  and  staff  as  anything 
different  than  a  passing  management  fad.”96  Likewise,  Moynihan  and  Lavertu’s  research 
for  the  Brookings  Institute  found  government-wide  reform  efforts  are  contingent  upon 
bureaucratic  behavior,  which  is  difficult  for  reformers  to  control  and  observe,  and 
ultimately  results  in  limited  successful  outcomes.97 

Perhaps  most  important  in  the  culture  change  is  the  deliberate  focus  on  using  the 
performance  management  system  to  drive  improved  performance  through  data-informed 
decisions.98  Asch  et  al.  found,  in  order  to  achieve  results  in  the  process,  a  culture  of 
accountability  must  be  clearly  identified  in  the  performance  management  system.  If 
performance  measures  are  to  have  any  impact  on  changing  an  organization’s  culture,  it 
must  be  clear  who  is  responsible  for  implementing  the  changes  implied  by  the  measure.99 
Moynihan  and  Lavertu’s  research  showed  the  ability  to  link  measures  to  actions  may 
depend  on  the  knowledge  and  skills  of  the  person  or  team  running  a  program.100  Training 
and  critical  thinking  skills,  in  turn,  can  help  managers  interpret  and  analyze  data. 101  This 
is  also  supported  by  Rosett  and  Rosett’s  research,  which  shows  that  those  who  implement 
a  perfonnance  management  system  should  have  substantial  training  in  scientific 
reasoning  and  should  understand  how  to  satisfy  their  customers.  In  addition,  to  facilitate 
process  improvement  efforts,  employees  may  need  some  cross-training  to  become 
familiar  with  processes  directly  related  to  their  functions.102  This  training  should  include 
indoctrination  to  the  organization’s  vision. 103  These  findings  are  also  evident  in 
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Donnelly’s  thesis,  which  found  the  self-assessment  system  used  by  DHS  to  evaluate 
SLTT  jurisdictions  lacks  methodological  validity  and  yields  imprecise  measurements.104 


D.  HISTORY  OF  DHS’S  RESPONSE  TO  THE  PERFORMANCE  MANDATE 

DHS  has  undertaken  a  number  of  initiatives  to  validate  expenditures  on  the 
various  emergency  preparedness  grant  programs.  Among  these  are  the  Capability 
Assessment  for  Readiness  (1997),  the  Gap  Analysis  Program  (2007),  the  State 
Preparedness  Report  (SPR)  (2008),  the  Pilot  Capability  Assessment  (2008),  the  Cost-to- 
Capability  Initiative  (2008),  National  Incident  Management  System  Compliance 
Assistance  Support  Tool  (NIMSCAST)  (2012-2013),  The  National  Preparedness  System 
(NPS)  (2006-2008),  and  the  Threat  Hazard  Identification  and  Risk  Assessment  (THIRA) 
(2012).  Today,  the  SPR  and  THIRA  remain  the  core  data  collection  tools  for  DHS  to 
assess  national  preparedness. 

The  GAO,  however,  notes  that  because  “clear,  objective,  and  quantifiable 
capability  requirements  and  perfonnance  measures”  have  yet  to  be  developed,  DHS’s 
efforts  to  assess  national  preparedness  will  continue  to  falter.105  Among  the  critiques  is 
the  flexibility  afforded  to  states  to  develop  individual  requirements  and  standards, 
preventing  analysis  that  could  identify  differences  and  capability  gaps. 106 

In  addition  to  the  systems  DHS  developed  and  implemented  over  the  years,  there 
are  a  number  of  policy  documents  that  guide  DHS  performance  measurement. 
Presidential  Policy  Directive  (PPD)-8  is  the  seminal  document  for  national  preparedness, 
which  directed  the  development  of  the  National  Preparedness  Goal,  the  National 
Preparedness  System,  and  the  National  Preparedness  Report.101  The  DHS  Strategic  Plan, 
another  key  document,  outlines  the  Department’s  mission,  vision,  goals  and 
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objectives.108  According  to  the  Strategic  Plan,  state,  tribal,  territorial,  and  local  entities 
will  play  a  significant  role  in  four  of  the  five  core  mission  areas:  “preventing  terrorism 
and  enhancing  security,  secure  and  manage  our  borders,  safeguard  and  secure  cyberspace, 
and  ensuring  resilience  to  disasters.”109  The  identified  goals  and  objectives  within  the 
DHS  Strategic  Plan  will  serve  as  the  basis  for  the  development  of  perfonnance  metrics. 

In  compliance  with  the  GPRA,  Homeland  Security  Presidential  Directive-8 
directed  DHS  to  “establish  measurable  readiness  priorities  and  targets  to  include 
readiness  metrics.”110  Additionally,  it  required  “standards  for  preparedness  assessments 
and  a  system  for  assessing  the  nation’s  overall  preparedness  for  responding  to  major 
events.”111  Public  Law  109-295  Section  649,  also  known  as  the  Post-Katrina  Emergency 
Management  Refonn  Act  (PKEMRA),  included  requirements  to  “establish  a 
comprehensive  system  to  assess,  on  an  ongoing  basis,  the  Nation’s  prevention 
capabilities  and  overall  preparedness,  including  operational  readiness.”112  PKEMRA  also 
requires  the  system  to  assess  all  related  plans  and  strategies  using  “clear  and  quantifiable 
performance  metrics,  measures,  and  outcomes.”113  The  assessments  outcomes  must  be 
reported  annually  and  will  include,  among  other  requirements,  the  overall  results  as  well 
as  the  resources  required  to  meet  jurisdictional  preparedness  goals  and  the  estimated 
costs. 114 
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E.  CONCLUSION 

Over  the  past  ten  years,  DHS  has  implemented  multiple  programs  designed  to 
capture  preparedness  data.  Those  programs  can  be  analyzed  and  compiled  into  a  national 
assessment  report  as  required  by  law. 

In  2015,  the  National  Preparedness  Report  data  consists  of  the  following  sources 
(reported  as  more  than  450  total)115: 

•  Open  source  materials 

•  Solicitation  for  data  from  federal  agencies 

•  Federal  agency,  department,  and  coordination  group  engagement  activities 

•  State  Preparedness  Reports  and  Threat  Hazard  Identification  and  Risk 
Assessments 

•  Applying  exercise,  funding,  and  assessment  criteria 

•  Collaboration  with  143  stakeholders 

Although  the  report  contains  many  local  success  stories  and  anecdotal  infonnation 
from  opinion  surveys,  it  does  not  identify  goals  with  measurable  objectives  or  charts 
showing  rate  of  progress.  There  is  also  little  connectivity  to  other  DHS  strategic  plans 
that  would  enable  them  to  produce  consistently  reliable  data.  Measures  should  be  focused 
on  the  critical  few  to  ensure  that  staff  members  are  not  overwhelmed  with  data  collection. 

Communication,  incentives,  and  culture  are  the  final  three  factors  essential  to 
establishing  a  successful  performance  management  system.  Management  plays  a  key  role 
in  each  of  these  areas  as  well.  Managers  and  leaders  at  all  levels  must  ensure  the 
organizational  vision,  priorities,  goals,  and  objectives  are  widely  shared  across  the 
organization.  This  will  promote  a  unified  effort  among  personnel,  allowing  them  to  work 
toward  the  vision.  Managers  can  also  ensure  staff  are  rewarded  for  meeting  or  exceeding 
interim  and  long-term  goals.  Although  such  a  reward  system  can  encourage  unintended 
behavior  and  outcomes  if  not  administered  properly,  perhaps  the  greatest  reward  is  a 


115  FEMA,  National  Preparedness  Report,  1. 
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performance  management  system  that  provides  value  to  the  staff  and  management,  and 
makes  individual  work  tasks  less  complicated  and  more  productive. 

Finally,  the  previously  discussed  factors  discussed — a  committed  management, 
effective  measures,  robust  strategic  planning,  valued  incentives,  and  widely  shared 
communications — are  necessary  for  an  organizational  culture  focused  on  performance 
results.  Managers  can  further  affect  an  organization’s  culture  by  ensuring  everyone  is 
held  accountable  for  performance  results,  while  also  giving  the  staff  the  necessary 
training  to  use  the  perfonnance  management  system  successfully.  Perhaps  the  greatest 
cultural  change  in  the  organization  will  be  managers’  use  of  data  and  information  to  make 
informed  decisions  at  all  levels. 

It  is  evident  by  the  review  of  the  literature  that  these  factors  can  facilitate  the 
implementation  of  a  perfonnance  management  system.  An  organization,  however,  must 
be  “all  in,”  ensuring  all  the  identified  factors  for  success  are  aligned  and  integrated  and 
not  succumb  to  the  common  check  box  approach  of  planning,  measuring,  and  reporting 
results  as  independent  and  isolated  activities.  It  will  require  strong  leadership, 
persistence,  and  time  to  change  organizational  culture.  There  is  still  much  work  left  to  do 
in  order  for  DHS  to  comply  with  the  GPRA-MA’s  intent. 

A  review  of  the  literature  also  found  no  significant  examples  of  highly  successful 
performance  management  systems  within  the  federal  government,  although  research 
indicates  the  implementation  of  such  a  system  can  yield  results.  But  what  are  the 
elements  of  a  perfonnance  management  system  that  could  produce  the  expected  results? 
Research  indicates  the  most  important  factor  in  developing  a  successful  performance 
management  system  is  management’s  role  and  interaction  with  the  entire  process. 
Mangers  at  all  levels  must  be  fully  committed  to  the  effort,  from  the  allocation  of 
resources  to  the  daily  use  of  performance  data  to  infonn  decisions.  Additionally, 
managers  play  an  essential  role  in  each  of  the  five  critical  factors  for  successful 
implementation:  strategic  planning,  measurement,  incentives,  communication,  and 
culture. 
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Managers  are  integral  to  setting  the  organization’s  vision,  and  for  establishing 
how  to  achieve  it.  They  must  set  an  action  plan  that  includes  organizationally  aligned 
goals  and  objectives,  and  they  are  accountable  for  allocating  proper  resources  for 
achieving  these  goals  and  maintaining  the  performance  management  system  once  it  is  in 
place.  Managers  must  also  ensure  the  performance  management  system  includes  metrics 
that  can  measure  progress  toward  the  strategic  vision.  In  order  to  provide  actionable  data, 
these  measures  require  certain  characteristics;  measures  should  consider  all  aspects  of  the 
organization’s  perfonnance,  to  include  structure,  process,  and  outcomes.  Outcome 
measures  are  considered  the  most  prescient  in  determining  progress.  These  measures, 
however,  generally  take  longer  to  compile  and  also  require  metrics  that  can  detennine  if 
the  actions  in  place  are  taking  the  organization  in  the  proper  direction  on  a  shorter 
interval.  In  addition,  the  research  indicates  measures  should  be  well  defined  and 
integrated,  and  should  not  succumb  to  the  common  “check  box”  approach  of  planning, 
measuring,  and  reporting  results  as  independent  and  isolated  activities.  Changing 
organizational  culture  will  require  strong  leadership,  persistence,  and  time. 
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III.  THE  PROBLEM  SPACE  DEFINED 


A.  INTRODUCTION 

The  problem  of  establishing  a  perfonnance  management  system  for  the  homeland 
security  enterprise  is  evaluated  through  three  central  components:  the  driving  forces  and 
requirements  for  a  perfonnance  management  system,  the  initiatives  taken  by  DHS  to 
implement  a  system,  and  these  efforts’  gaps  and  shortfalls.  There  are  three  main 
document  categories  influencing  the  design  and  implementation  of  a  perfonnance 
management  system  for  grant  programs.  They  are:  public  laws  and  federal  regulations 
(which  mandate  the  specific  system  requirements),  DHS  policy  documents,  and  DHS 
strategy  documents.  In  addition  to  the  various  assessment  tools  developed  and 
implemented  over  the  past  10  years,  numerous  policy,  doctrine,  and  legislation  efforts 
have  also  been  enacted. 

B.  PUBLIC  LAW  AND  FEDERAL  REGULATION 

1.  The  Robert  T.  Stafford  Disaster  Relief  and  Emergency  Assistance  Act 
of  1974 

The  Stafford  Act  established  “programs  for  the  federal  government  to  provide 
major  disaster  and  emergency  assistance  to  states,  local  governments,  tribal  nations, 
individuals,  and  qualified  nonprofit  organizations.”116  Recently  updated  under  the  Sandy 
Recovery  Improvement  Act,  the  Stafford  Act  is  the  foundational  guidance  for  all  FEMA 
programs. 

While  the  Stafford  Act  offers  little  guidance  for  performance  measurement,  it 
holds  the  president  responsible  for  establishing  comprehensive  standards  to  assess  the 
effectiveness  of  federal  major  disaster  and  emergency  assistance  programs.  In  addition, 
annual  reviews  of  major  disaster  and  emergency  preparedness  at  the  federal  and  local 
levels  will  be  conducted  to  assure  programs’  maximum  coordination  and  effectiveness,  as 


116  Robert  T.  Stafford  Disaster  Relief  and  Emergency  Assistance  Act,  Pub  L.  93-288,  as  amended,  42 
U.S.C.  §5121  (2013) 
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well  as  their  consistency  in  policies  for  reimbursement  of  States.117  Finally,  under 
Section  318,  “the  president  is  authorized  to  conduct  audits  of  any  assistance  provided  to 
state  and  local  governments  by  the  Stafford  Act,  to  assure  compliance  with  related 
regulations.”118 

Although  disaster  grant  management  and  perfonnance  is  not  the  focus  of  this 
research,  there  is  benefit  to  be  gained  from  collecting  data  and  analyzing  the  performance 
of  disaster  response  and  recovery  activities.  The  ability  to  analyze  data  on  federal,  state, 
and  local  efforts  under  the  three  types  of  federal  disaster  declarations  (Fire  Management 
Assistance  Grants,  emergencies,  and  major  declarations)  would  inform  the  homeland 
security  enterprise  on  a  jurisdiction’s  level  of  preparedness,  and  highlight  capability  gaps. 
This  data  would  be  instrumental  in  an  overall  performance  management  system  because 
it  measures  outcomes  of  a  variety  of  programs  and  processes.  However,  it  is  difficult  to 
dedicate  resources  to  track  disaster  response  performance  data  during  an  incident  due  to 
resource  constraints  throughout  the  homeland  security  enterprise.  There  may  be 
opportunities  for  collecting  data  from  incident  command  system  standard  forms,  archived 
briefings,  and  the  emergency  operations  center  incident  management  platform  during  the 
recovery  phase  or  following  the  incident. 

2.  Government  Performance  and  Results  Act  of  1993 

The  Government  Performance  and  Results  Act  (GPRA)  was  enacted,  in  part,  to 
improve  taxpayer  confidence  in  the  government.  Specifically,  the  act  examined  the 
government’s  capability  “to  achieve  program  results,  improve  federal  program 
effectiveness  and  accountability,  provide  a  framework  for  federal  managers  to  focus  on 
program  results,  service  quality  and  customer  satisfaction.”119  In  addition,  the  GPRA 
served  as  a  mechanism  to  inform  congressional  committees  on  forming  budget 
allocations  to  various  programs  based  on  results  or  outcome  data. 


117  Ibid. 

118  Ibid. 

119  Government  Performance  and  Results  Act  of  1993,  Pub.  L.  No.  103-62,  107  Stat.  285  (1993) 
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To  accomplish  these  goals,  the  GPRA  “established  strategic  planning, 
performance  planning,  and  performance  reporting  for  agencies  to  communicate  progress 
toward  achieving  their  missions.”120  The  strategic  plan  is  required  for  primary  functions 
and  operations,121  and  the  perfonnance  plan  is  required  for  each  program  activity  and 
must  be  connected  to  the  agency’s  strategic  plan.122  Finally,  the  annual  report  highlights 
each  program’s  actual  performance  compared  to  the  stated  goals  for  Congress  and  the 
president.123 


3.  Homeland  Security  Act  of  2002 

The  Homeland  Security  Act  established  DHS  from  the  consolidation  of  almost 
two  dozen  federal  agencies  with  similar  mission  goals  under  a  single  department  head. 
The  new  department  would  serve  as  the  focal  point  to  plan  and  coordinate  activities 
necessary  to  protect  the  homeland  against  all  natural  or  man-caused  disasters.124  The  Act 
also  directed  compliance  with  GPRA  requirements  by  requiring  each  agency  to  provide 
an  annual  report  and  follow  program  performance  requirements.125 

4.  Homeland  Security  Presidential  Directive  5,  Management  of  Domestic 
Incidents,  2003 

Homeland  Security  Presidential  Directive  5  (HSPD-5)  established  clear  command 
and  control  for  domestic  incidents  under  the  Secretary  of  Homeland  Security.  It  also  set 
guidelines  for  the  individual  agency  actions  during  an  emergency  “to  prepare  for,  respond 
to,  and  recover  from  terrorist  attacks,  major  disasters,  and  other  emergencies.”126  Finally, 


120  OMB,  Circular  No.  A-ll,  Preparation,  Submission,  and  Execution  of  the  Budget  (Washington, 
DC:  The  White  House,  2014),  Part  6,  Section  200.4,  http://www.whitehouse.gov/sites/default/ 
files/omb/assets/al  l_current_year/al  l_2014.pdf. 

121  Government  Performance  and  Results  Act  of  1993,  Stat.  286. 

122  Ibid.,  Stat.  287. 

123  Ibid.,  107  Stat.  288. 

124  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  86. 

125  Homeland  Security  Act  of 2002,  Pub  L.  No.  107-296,  116  Stat.  2264  (2002) 

126  Bush,  Homeland  Security  Presidential  Directive  5. 
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HSPD-5  formally  established  a  national  incident  management  system  (NIMS)  and  a 
national  response  plan  (NRP)  each  required  in  the  Homeland  Security  Act. 

NIMS  provides  standards  to  facilitate  coordination  and  effectiveness  for  any 
entity,  public  or  private,  conducting  operations  during  a  domestic  incident.127 
Additionally,  the  adoption  of  NIMS  is  required  before  any  federal  or  state  agency  can 
receive  federal  funds  for  preparedness  development.128 

5.  Homeland  Security  Presidential  Directive  8,  National  Preparedness, 
2003 

Homeland  Security  Presidential  Directive  8  (HSPD-8)  established  the  national 
preparedness  goal,  which  included  the  development  of  a  system  to  collect  and  analyze 
data  across  the  homeland  security  enterprise  as  allowed  by  law.  The  system’s  overall  goal 
is  to  provide  an  annual  assessment  of  the  Nation’s  readiness  and  included  requirements  to 
establish  measurable  priorities  as  well  as  specific  metrics  and  assessment  standards.129 

HSPD-8  also  called  for  ah  states  to  develop  strategies  that  would  prioritize 
requirements  and  assess  program  effectiveness.  Although  the  strategies  were  to 
encompass  an  all-hazard  approach,  federal  assistance  to  state  and  local  entities  remained 
terrorism  centric.130 

6.  Post-Katrina  Emergency  Management  Reform  Act  of  2006 

The  Post  Katrina  Emergency  Management  Refonn  Act  (PKEMRA)  required 
FEMA  establish  a  National  Preparedness  System.  PKEMRA  assigned  eight  components 
to  the  National  Preparedness  System  to  facilitate  meeting  the  targets  established  in  the 
National  Preparedness  Goal  (the  Goal)131  (Section  C,  Strategy  Documents,  Policy  and 
Guidance,  contains  a  more  detailed  discussion  of  The  Goal).  Of  the  eight  components, 

127  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  89,  Table  4. 

128  Department  of  Homeland  Security,  Funding  Opportunity’  Announcement,  34. 

129  Bush,  Homeland  Security  Presidential  Directive  5. 

130  Ibid. 

131  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
Management  Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1425  (2006). 
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three  have  applicability  to  this  research:  target  capabilities  and  preparedness  priorities,  a 
comprehensive  assessment  system,  and  reporting  requirements. 

The  target  capabilities  were  to  serve  as  “guidelines  to  define  risk-based  target 
capabilities  for  federal,  state,  local  and  tribal  government  preparedness.”  The  guidelines 
are  to  be  specific,  flexible,  and  measureable.  Measuring  current  capability  levels  to 
compare  against  the  established  targets  will  identify  the  resources  needed  to  attain  the 
target  levels  of  capability.  The  preparedness  priorities  should  use  a  risk-based  approach 
that  will  “appropriately  balance  the  risk  of  all  hazards  with  the  resources  required  to 
prevent,  respond  to,  recover  from,  and  mitigate  the  hazards.”132 

The  National  Preparedness  System  includes  the  task  of  establishing  a 
comprehensive  assessment  system.  The  assessment  system  is  mandated  to  measure 
performance  and  outcomes  in  meeting  established  target  capabilities,  the  resources 
needed  to  achieve  the  designated  target,  and  the  perfonnance  of  agencies  and 
jurisdictions  in  the  course  of  conducting  actual  operations  as  well  as  during  exercises  and 
training  events. 133 

The  act  also  required  that  FEMA  annually  report  on  the  Nation’s  level  of 
preparedness.  The  report  would  include  information  collected  in  the  comprehensive 
assessment  system  along  with  an  evaluation  of  federal  resources  and  their  capabilities 
available  to  respond  to  disasters  and  finally  an  estimate  of  the  outlay  needed  to  fund 
SLTT  preparedness  priorities.134  To  inform  the  federal  preparedness  report,  states  are 
also  required  to  submit  an  annual  report.  State  assessments  would  include  overall 
compliance  with  federal  plans  and  strategies,  a  current  capability  assessment  and  an 
estimate  of  funding  needed  to  meet  preparedness  goals. 135 

The  PKEMRA  bolstered  FEMA’s  status  as  a  federal  agency  within  DHS  by 
designating  the  Administrator  as  the  President’s  principal  advisor  for  all  emergency 


132  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
Management  Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1426  (2006). 

133  Department  of  Homeland  Security  Appropriations  Act  2007,  1428. 

134  Department  of  Homeland  Security  Appropriations  Act  2007,  1429. 

135  Department  ofHomeland  Security  Appropriations  Act  2007,  1430. 
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management  matters  and  could  be  designated  a  Cabinet  member  during  a  disaster.136  In 
addition,  the  PKEMRA  restored  to  FEMA  the  Directorate  of  Preparedness  and  the 
functions  of  emergency  alert,  continuity,  and  grant  management  programs.137 

7.  GPRA  Modernization  Act  of  2010 

The  1993  GPRA  was  updated  via  the  GPRA  Modernization  Act  (GPRAMA)138 
and  remains  the  federal  government’s  principal  perfonnance  management  guidance.  It 
provided  additional  specificity  to  the  strategic  and  perfonnance-planning  requirements 
for  affected  departments.139  In  addition,  the  revisions  increased  agency  consultation  with 
congressional  committees  in  developing  and  updating  goals  and  objectives.  Consideration 
was  also  given  to  external  environmental  factors  affecting  agency  performance,  along 
with  the  interaction  and  collaboration  with  other  agencies  to  achieve  performance  goals 
and  objectives.140  The  stated  purposes  of  the  GPRAMA  are  to: 

•  Improve  the  confidence  of  the  American  people  in  the  capability  of  the 
federal  government,  by  systematically  holding  federal  agencies 
accountable  for  achieving  program  results 

•  Improve  program  performance  by  requiring  agencies  to  set  goals,  measure 
performance  against  those  goals  and  report  publicly  on  progress 

•  Improve  federal  program  effectiveness  and  public  accountability  by 
promoting  a  focus  on  results,  service  quality  and  customer  satisfaction 

•  Help  federal  managers  improve  service  delivery,  by  requiring  that  they 
plan  for  meeting  program  goals  and  by  providing  them  with  information 
about  program  results  and  service  quality 

•  Improve  congressional  decision-making  by  providing  more  information  on 
achieving  statutory  objectives  and  on  the  relative  effectiveness  and 
efficiency  of  federal  programs  and  spending 

•  Improve  internal  management  of  the  federal  government,  and 


136  Department  of  Homeland  Security  Appropriations  Act  2007,  1398. 

137  Department  of  Homeland  Security  Appropriations  Act  2007,  1400. 

138  Government  Performance  and  Results  Act  (GPRA)  Modernization  Act  of  2010,  Pub.  L.  No.  Ill— 
352,  124  Stat.  3866  (2011). 

139  Ibid. 

140  Ibid. 
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•  Improve  usefulness  of  perfonnance  and  program  information  by 
modernizing  public  reporting. 141 

The  GPRAMA  is  the  new  foundation  for  developing  more  effective  and  efficient 
government  agencies  by  creating  an  approach  that  allows  managers  and  leaders  to  make 
informed  decisions  and  set  policy  based  on  data  and  analysis. 

8.  Code  of  Federal  Regulations  Title  44,  Emergency  Management  and 
Assistance 

Code  of  Federal  Regulations  (CFR)  44  provides  the  principle  set  of  rules  and 
regulations  governing  emergency  management  and  assistance.142  The  regulation’s  stated 
central  purpose  is  to  “foster  intergovernmental  partnership  and  strengthen  federalism  by 
connecting  state,  local,  tribal,  and  territorial  (SLTT)  jurisdictions  into  the  preparedness 
system.”143 

CFR  44  underscores  the  monitoring  and  program  performance  requirements  for 
all  FEMA  grantees.  Grantees  are  responsible  for  complying  with  all  applicable  federal 
requirements  and  ensuring  they  meet  established  performance  goals.  Grantees  are  also 
responsible  for  submitting  annual  perfonnance  reports  that  compare  actual 
accomplishments  to  the  established  objectives  and  quantify  output  analysis  if 
appropriate.144  In  addition  to  the  annual  perfonnance  reports,  a  closeout  report  is  required 
when  the  federal  agency  determines  all  required  grant  work  has  been  completed.145 
Finally,  the  federal  agency  may  consider  a  subgrantee  “high-risk”  if  there  is  a  history  of 
unsatisfactory  performance  due  to  cost  overruns  or  failure  to  meet  program  objectives. 
Under  these  circumstances,  the  grantor  may  restrict  grant  fund  distribution,  and  require 
additional  and  more  detailed  reporting  and  increased  monitoring. 146 


141  OMB,  Circular  No.  A-ll,  Preparation,  Submission,  and  Execution  of  the  Budget  (Washington, 
DC:  The  White  House,  2014),  Part  6,  Section  200.4, 

http://www.whitehouse.gov/sites/default/files/omb/assets/all_current_year/all_2014.pdf. 

142  Emergency  Management  and  Assistance,  44  U.S.C.  §1-499,  (2013). 

143  Ibid. 

144  Ibid. 

145  Ibid. 

146  Ibid. 
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9. 


Recommendations  of  the  9/11  Commission  Act  of  2007 


The  9/11  Commission  Act  implemented  recommendations  made  by  the  9/11 
Commission,  and  strengthened  the  incident  command  systems’  use.  The  act  (Pub.  L.  No. 
110-53)  also  created  additional  block  grant  funding  for  state,  local,  and  tribal 
governments  under  the  Homeland  Security  Grant  Program.  This  program  established  four 
new  grant  programs  to  counter  the  terror  threat:  the  Urban  Area  Security  Initiative,  the 
State  Homeland  Security  Grant  Program,  the  Law  Enforcement  Terrorism  Prevention 
Program  and  the  Tribal  Homeland  Security  Grant  Program. 147 

The  act  also  provided  accountability  requirements  for  all  new  and  existing 
recipients  of  grant  funding.  The  first  mandate  was  an  agency  programmatic  and  financial 
review  to  be  conducted  by  DHS  at  least  once  every  two  years.  This  review  will  ensure 
grant  funds  were  used  according  to  all  applicable  laws,  guidance,  and  plans,  and  will 
examine  how  the  funds  enhanced  grantee  capabilities.148  DHS  is  also  mandated  to 
conduct  a  performance  assessment  (as  established  under  PKEMRA)  to  ensure  grant  funds 
are  appropriately  used  to  meet  established  target  capabilities  and  preparedness 
priorities.149  The  second  mandate  was  an  Office  of  Inspector  General  Perfonnance  audit 
to  ensure  the  effective  and  appropriate  use  of  grant  funds.  All  states  receiving  federal 
grant  funds  receive  at  least  one  audit  every  seven  years.  In  addition,  sample  audits  are 
conducted  based  on  the  size  of  the  grant  and  the  recipient’s  past  perfonnance  history. 150 

10.  Presidential  Policy  Directive  8,  National  Preparedness  2011 

Presidential  Policy  Directive-8  (PPD-8)  is  the  seminal  document  for  national 
preparedness,  directing  the  development  of  four  key  parts:  a  National  Preparedness  Goal; 
National  Preparedness  System  (NPS);  a  series  of  integrated  frameworks  covering  each  of 
the  five  core  mission  areas  (prevention,  protection,  mitigation,  response,  and  recovery); 
and  a  National  Preparedness  Report  (NPR).  PPD-8’s  purpose  was  to  “strengthen  the 

147  Implementing  Recommendations  of  the  9/11  Commission  Act  of  2007,  Pub.  L.  No.  110-53,  121 
Stat.  274-281  (2007). 

148  Ibid. 

149  Ibid. 

150  Ibid. 
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security  and  resilience  of  the  United  States  through  systematic  preparation  for  the  threats 
that  pose  the  greatest  risk  to  the  security  of  the  Nation.”  PPD-8  specifies  that  the 
responsibility  for  national  preparedness  resides  with  all  levels  of  government,  the  private 
and  nonprofit  sectors,  and  individuals,  thus  promoting  the  concepts  of  “all  of  Nation”  and 
“whole  community”  prevalent  throughout  DHS  guidance  documents.151  Finally,  PPD-8 
directs  a  comprehensive  approach  “to  measure  the  operational  readiness  of  national 
capabilities  with  clear,  objective,  and  quantifiable  perfonnance  measures.”152 

Furthermore,  PPD-8  directed  the  NPS  to:  build  and  sustain  capabilities;  include  a 
system  of  integrated  planning  frameworks;  provide  resource  and  equipment  guidance; 
support  community  and  business  preparedness  planning  efforts;  and  develop  a 
methodology  to  measure  operational  readiness  with  clear,  objective  and  quantifiable 
performance  measures.153 

C.  STRATEGY  DOCUMENTS,  POLICY  AND  GUIDANCE 

1.  National  Strategy  for  Homeland  Security  of  2007 

The  stated  purpose  of  the  National  Strategy  for  Homeland  Security  “is  to  guide, 
organize,  and  unify  homeland  our  Nation’s  homeland  security  efforts.  It  provides  a 
common  framework  by  which  our  entire  Nation  should  focus  its  effort.”154  One  of  the 
tenants  within  the  strategy  is  the  delineation  of  national  response  roles  and 
responsibilities.  The  principle  of  disaster  response  is  scalability  but  that  “all  incidents 
should  be  handled  at  the  lowest  jurisdictional  level  possible.”155 


151  Barack  Obama,  Presidential  Policy  Directive  8:  National  Preparedness  (Washington,  DC:  The 
White  House,  2011),  1,  http://www.dhs.gov/presidential-policy-directive-8-nationalpreparedness. 

152  Obama,  Presidential  Policy  Directive  8,  4-5. 

153  Ibid.,  2-4. 

154  Homeland  Security  Council,  National  Homeland  Security  Strategy’,  (Washington,  DC:  The  White 
House,  2007),  1.  http://www.dhs.gov/xlibrary/assets/nat_strat_homelandsecurity_2007.pdf. 

155  Homeland  Security  Council,  National  Homeland  Security  Strategy’,  (Washington,  DC:  The  White 
House,  2007),  33.  http://www.dhs.gov/xlibrary/assets/nat_strat_homelandsecurity_2007.pdf. 
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The  strategy  supports  the  role  of  the  community  and  individual  by  identifying  the 
need  to  create  a  culture  of  preparedness.  “Citizens,  communities,  the  private  sector  and 
non-profit  organizations  each  perform  a  central  role  in  homeland  security”156 

Finally,  the  strategy  calls  for  a  Homeland  Security  Management  System  led  by 
federal  government  agencies  but  is  dependent  on  SLTT  jurisdictions  for  ultimate  success. 
The  management  system  is  comprised  of  four  distinct  phases:  1)  guidance,  to  include 
plans,  policies,  guidance,  and  doctrine;  2)  planning,  to  include  strategic,  operational,  and 
tactical;  3)  execution,  through  operational  response  or  exercises  and;  4)  assessment  and 
evaluation,  to  identify  both  successful  practices  and  areas  for  improvement  that  will 
inform  potential  changes  to  future  guidance  and  plans.157 

2.  National  Preparedness  Guidelines  (NPG)  Sept  2007 

The  vision  of  the  NPG  supports  the  collaboration  among  all  levels  of  government 
and  stresses  the  importance  of  the  private  sector,  citizens,  and  nongovernmental 
organizations.158  The  guidelines  establish  a  framework  of  plans,  strategies,  and  systems 
into  the  National  Preparedness  System  and  include  a  risk-based,  all-hazards,  and 
capabilities-based  approach  to  preparedness.159  A  capability  is  defined  as  “the 
combination  of  elements  required  to  deliver  the  desired  outcome.”160  The  NPG  offers  37 
capabilities  within  the  primary  mission  areas  of  prevention,  response,  protection,  and 
recovery  and  includes  five  common  capabilities  across  all  mission  areas.  A  desired 
outcome  is  defined  for  each  capability  as  a  means  to  begin  the  measurement  and 
assessment  process.161  The  NPG  outlined  a  process  to  include  standards  and  specific 


156  Homeland  Security  Council,  National  Homeland  Security  Strategy,  (Washington,  DC:  The  White 
House,  2007),  42.  http://www.dhs.gov/xlibrary/assets/nat_strat_homelandsecurity_2007.pdf. 

157  Homeland  Security  Council,  National  Homeland  Security  Strategy,  (Washington,  DC:  The  White 
House,  2007),  44-45.  http://www.dhs.gov/xlibrary/assets/nat_strat_homelandsecurity_2007.pdf. 

158  Department  of  Homeland  Security,  National  Preparedness  Guidelines  (Washington,  DC,  2007),  1. 

159  Ibid.,  2-3. 

160  Ibid.,  5. 

161  Ibid.,  4-10. 
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metrics  were  in  development  at  the  time  of  publication  but  it  never  “established 
measurable  capability  requirements  for  federal  and  SLTT  preparedness  efforts.”162 


3.  National  Preparedness  Goal  of  2011 

The  National  Preparedness  Goal  (the  Goal)  was  the  first  document  released  in 
support  of  PPD-8.  The  Goal  consists  of  a  vision,  capabilities,  and  priorities  for  national 
preparedness.163 

The  Goal  is  described  as  “the  Nation’s  approach  to  preparing  for  the  threats  and 
hazards  that  pose  the  greatest  risk  to  the  security  of  the  United  States”  and  defines 
success  as  “a  secure  and  resilient  Nation  with  the  capabilities  required  across  the  whole 
community  to  prevent,  protect  against,  mitigate,  respond  to,  and  recover  from  the  threats 
and  hazards  that  pose  the  greatest  risk.”164  The  Goal  contains  31  core  capabilities  that  are 
described  as  the  essential  elements  necessary  to  successfully  execute  the  “five  core 
mission  areas  of:  prevention,  protection,  mitigation,  response,  and  recovery.”165  Of  the 
3 1  capabilities,  three  are  common  to  all  five  mission  areas,  while  the  remaining  28  are 
specifically  aligned  to  one  or  more  mission  areas.166  Each  core  capability  has  one  or 
more  capability  targets  identified  as  the  performance  threshold(s)  for  each  core 
capability.  These  target  capabilities  are  the  basis  for  developing  future  perfonnance 
metrics  that  will  be  used  to  track  national  preparedness  progress.167  These  core 
capabilities  replaced  the  Target  Capabilities  List  established  in  2007. 


162  GAO,  FEMA  Has  Made  Limited  Progress,  3 1 . 

163  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  89,  Table  4. 

164  FEMA,  National  Preparedness  Goal  (Washington,  DC:  Department  of  Homeland  Security,  2011), 
1,  http://www.fema.gov/media-library-data/20130726-1828-25045- 
9470/national_preparedness_goal_201  l.pdf. 

165  FEMA,  National  Preparedness  Goal  (Washington,  DC:  Department  of  Home  land  Security,  2011), 
1,  http://www.fema.gov/media-library-data/20130726-1828-25045- 
9470/national_preparedness_goal_201  l.pdf. 

166  FEMA,  National  Preparedness  Goal,  2,  Table  1. 

167  FEMA,  National  Preparedness  Goal  (Washington,  DC:  Department  of  Home  land  Security,  2011), 
1,  http://www.fema.gov/media-library-data/20130726-1828-25045- 
9470/national_preparedness_goal_201  l.pdf. 
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4.  National  Preparedness  System,  2011 

The  NPS  is  intended  to  be  an  organized  process  for  the  whole  community  to 
conduct  preparedness  activities  and  achieve  the  NPG.  The  NPS  currently  has  six  parts: 

•  Identifying  and  Assessing  Risk — the  collection  of  information  to  identify 
threats  and  hazards  along  with  their  respective  impacts.  The  national 
assessment  is  based  on  the  combination  of  the  Threat  Hazard 
Identification  and  Risk  Assessment  (THIRA)  conducted  by  SLTT 
jurisdictions  and  the  Strategic  National  Risk  Assessment  conducted  by 
federal  agencies 

•  Estimating  Capability  Requirements — developed  through  planning 
factors  that  use  the  threat  data  to  detennine  specific  resources  needed  to 
develop  the  capabilities  necessary  to  achieve  the  stated  outcome. 

•  Building  and  Sustaining  Capabilities — planning  to  prioritize  the 
identified  capability  gaps  and  sustaining  existing  capabilities. 

•  Planning  to  Deliver  Capabilities — coordination  and  involvement  with 
the  whole  community. 

•  Validating  Capabilities — exercise  and  test  plans  and  capabilities  to  test 
gaps  in  preparedness  and  measure  progress. 

•  Reviewing  and  Updating — a  regularly  scheduled  review  and  update  of 
jurisdictional  capabilities,  resources  and  plans.168 

The  NPS  attempts  to  integrate  preparedness  activities  that  have  already  been 
accomplished  into  a  system  that  brings  in  new  tools  and  processes  to  achieve  necessary 
outcomes  during  emergencies  and  disasters.  The  six-step  methodology  builds  on 
PKEMRA  by  enabling  states  to  collaborate  across  the  whole  community  for  national 
preparedness.169  The  NPS  validates  the  capability  needed  by  a  risk  assessment,  which 
informs  new  or  current  capabilities.  It  guides  states  to  develop  a  collaborative,  holistic 
strategy.  Each  state  is  mandated  to  aggregate  their  own  Threat  and  Hazard  Identification 
and  Risk  Assessment  (THIRA)  to  the  regional  and  then  national  level. 


168  FEMA,  National  Preparedness  Goal  (Washington,  DC:  Department  of  Homeland  Security,  2011), 
2-6,  http://www.fema.gov/media-library-data/20130726-1828-25045- 
9470/national_preparedness_goal_2011.pdf. 

169  Ibid. 
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5. 


Comprehensive  Preparedness  Guide  201  (The  Threat  Hazard 
Identification  and  Risk  Assessment),  2013 


The  Threat  Hazard  Identification  and  Risk  Assessment  (THIRA)  describes  a 
“standard  process  for  identifying  community-specific  threats  and  hazards  and  setting 
targets  for  each  core  capability,”  and  “includes  estimation  of  resources  needed  to  meet 
the  capability  targets.”170  The  THIRA  identifies  the  specific  “natural,  technological, 
accidental  and  malicious  causes”171  of  disasters  for  each  state  and  territory,  and  specifies 
established  capabilities  to  mitigate  or  respond  to  the  hazard,  as  well  as  the  shortfalls. 
These  shortfalls  or  gaps  should  be  linked  to  the  jurisdiction’s  funding  goals  and 
objectives.  However,  it  is  impractical  for  every  state  and  territory  to  counter  every  known 
threat  or  hazard.  Therefore,  the  risk  or  probability  of  the  threat  occurring  is  also  critical  to 
prioritizing  the  desired  capability  development. 

The  THIRA  is  a  relatively  new  DHS  requirement.  “FEMA  officials  stated  that  the 
THIRA  process  is  intended  to  facilitate  the  development  of  a  set  of  national  capability 
performance  requirements  and  measures.”172  “Such  measures,  however,  are  not  yet 
developed.  Until  clear,  objective,  and  quantifiable  capability  requirements  and 
performance  measures  are  developed,  the  NPS  will  not  be  able  to  identify  and  close  its 
resource  and  capability  gaps.”173 

6.  Strategic  and  Performance  Plans 

a.  Strategic  Plans 

Required  by  the  GPRAMA,  strategic  plans  require  updates  at  least  every  four 
years,  with  congressional  consultation  every  two  years.  The  extracted  elements  that 
should  be  included  in  the  strategic  plan  include: 


170  Department  of  Homeland  Security,  Threat  and  Hazard  Identification  and  Risk  Assessment  Guide, 
Comprehensive  Preparedness  Guide  (CPG)  201,  Second  Edition.  (Washington,  DC,  2013),  1, 
http://www.fema.gov/media-library- 

data/8ca0a9e54dc8b037a55b402b2a269e94/CPG20  l_htirag_2nd_edition.pdf. 

171  Robert  Bach  (thesis  advisor),  in  discussion  with  the  author,  February  13,  2015. 

172  Maurer,  National  Preparedness,  12. 

173  Ibid.,  13. 
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•  A  comprehensive  mission  statement  covering  the  major  functions  and 
operations  of  the  agency 

•  General  goals  and  objectives,  including  outcome-oriented  goals,  for  the 
major  functions  and  operations  of  the  agency 

•  A  description  of  how  to  achieve  the  goals  and  objectives,  including 
operational  processes  and  resources  required  and  cross  agency 
coordination 

•  Identification  of  the  key  external  factors  that  could  significantly  affect  the 
achievement  of  the  general  goals  and  objectives 

•  Establishment  of  a  process  to  evaluate  program  performance174 

DHS’s  2014-2018  Strategic  Plan  maintains  a  “five-mission  structure:  (1)  prevent 
terrorism  and  enhance  security,  (2)  secure  and  manage  our  borders,  (3)  enforce  and 
administer  our  immigration  laws,  (4)  safeguard  and  secure  cyberspace,  and  (5)  strengthen 
national  preparedness  and  resilience.”175  The  two  primary  identified  factors  are  unity  of 
effort  and  stakeholder  engagement.176  In  addition,  DHS  established  a  four-year  plan  to 
improve  its  ability  to  collect,  analyze,  and  report  on  data  sources,  referred  to  as  the 
Analytic  Agenda.  The  outcome  of  this  effort  aims  to  improve  DHS’s  ability  to  employ 
analytically  informed  decisions  and  operate  under  data-driven  management.177 

The  strategic  plan  should  be  the  seminal  document  to  answer  the  questions:  What 
does  it  mean  to  be  prepared?  What  are  the  attributes  of  a  prepared  homeland  security 
enterprise?  How  much  preparedness  is  enough?  The  biggest  challenge  for  determining 
preparedness  in  a  dynamic  and  changing  environment  is  that  we  often  do  not  know  what 
preparedness  is,  but  we  know  what  it  is  not  when  it  fails  to  meet  customer 
expectations.178  This  leads  to  perhaps  the  even  more  difficult  questions:  What  do  we 


174  Government  Performance  and  Results  Act  (GPRA)  Modernization  Act  of2010.  Pub.  L.  No.  111- 
352,  124  Stat.  3866-3867  (2011). 

175  Department  of  Homeland  Security  (DHS),  Fiscal  Years  2014-2018  Strategic  Plan  (Washington, 
DC:  Department  ofHomeland  Security,  2014),  6,  http://www.dhs.gov/sites/default/files/publications/FY14- 
18%20Strategic%20Plan.PDF. 

176  Ibid.,  6-7. 

177  DHS,  Fiscal  Years  2014-2018  Strategic  Plan,  11. 

178  Todd  Harrison,  “Rethinking  Readiness,”  Strategic  Studies  Quarterly  8  (Fall  2014):38. 
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need  to  be  prepared  for?  How  long  do  we  have  to  be  prepared — what  are  the  short  and 
long  term  issues?  What  agencies  or  capabilities  are  the  first,  second,  and  third  priorities 
for  funding?  “The  answers  to  all  of  these  questions  are  fundamentally  matters  of  strategy; 
what  it  means  to  be  “prepared”  can  only  be  understood  in  the  context  of  one’s 
strategy.”179 

The  challenges  of  overcoming  dynamic  and  evolving  threats  and  hazards  are 
great.  Effective  and  efficient  solutions  to  these  challenges  are  best  met  by  setting 
priorities:  “What  does  the  community  need  to  prepare  for?  What  resources  are  required  in 
order  to  be  prepared?  What  action  could  be  employed  to  lessen  or  eliminate  the 
threat?”180  For  these  reasons,  good  strategic  planning — defining  the  organization’s 
direction  for  the  future181 — is  critical.182  “The  complexity,  difficulty,  and  importance  of 
the  preparedness  system  underscore  the  importance  of  a  strategic  plan  in  the 
development,  implementation,  and  integration  of  the  system.”183 

b.  Performance  Plans 

The  perfonnance  plan’s  purpose  is  to  establish  intennediate  goals  for  the  current 
and  next  fiscal  year  that  will  move  the  organization  closer  to  meeting  their  strategic 
objectives.  The  relevant  elements  of  the  performance  plan  are: 

•  Identify  the  program  activities,  regulations,  expenditures,  policies,  and 
other  activities  contributing  to  each  perfonnance  goal 

•  Identify  a  lead  official  for  each  performance  goal,  responsible  for 
coordinating  the  efforts  to  achieve  the  goal 


179  Ibid.,  39. 

180  Department  of  Homeland  Security,  Threat  and  Hazard  Identification  and  Risk  Assessment  Guide, 
Comprehensive  Preparedness  Guide  (CPG)  201,  Second  Edition.  (Washington,  DC:  Department  of 
Homeland  Security,  2013)  4,  http://www.fema.gov/media-library- 
data/8ca0a9e54dc8b037a55b402b2a269e94/CPG20  l_htirag_2nd_edition.pdf. 

181  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  Air  Force  Publication  90-201 
(Washington,  DC:  Secretary  of  Air  Force  (IG),  2013),  38. 

182  Henry  H.  Wills,  Strengthening  Strategic  Planning  and  Management  at  DHS,  (RAND  Document 
No.  CT-387)  (RAND  Office  of  External  Affairs,  2013),  4. 
http://www.rand.org/pubs/testimonies/CT387.html. 

183  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  12, 
http://www.gao.gov/assets/290/289285.pdf. 
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•  Establish  performance  indicators  with  quarterly  targets  to  assess  overall 
progress 

•  Identify  major  management  challenges  toward  meeting  goals  and 
objectives 

•  Express  goals  in  an  objective,  quantifiable,  and  measurable  fonn 

•  Identify  the  program  activities,  regulations,  policies,  and  other  activities 
that  contribute  to  each  performance  goal,  both  within  and  external  to  the 
agency 

•  Identify  the  officials  responsible  for  each  performance  goal 

•  Establish  a  balanced  set  of  performance  indicators  to  be  used  in  measuring 
or  assessing  progress  toward  each  perfonnance  goal,  including  customer 
service,  efficiency,  output,  and  outcome  indicators 

•  Compare  actual  program  results  with  the  established  perfonnance  goals184 

c.  Analysis 

The  DHS  Strategic  Plan  for  Fiscal  Year  (FY)  2014-2018  provides  essential 
information  to  develop  a  perfonnance  measurement  system  that  can  effectively  inform 
decision-makers  to  shape  resource  allocations  and  detennine  progress.  Performance 
measures  are  defined  as  the  “metrics  used  to  ascertain  actual  perfonnance  against  target 
levels  identified  for  each  core  capability;  by  design,  they  are  clear,  objective,  and 
quantifiable.”185  The  strategic  plan  communicates  the  department’s  vision,  mission, 
goals,  and  objectives,  and  illustrates  selective  performance  measures  with  their  planned 
targets  over  a  four-year  period. 

The  FY  2012-2014  Annual  Performance  Report  (APR)  provides  a  full  list  of 
performance  measures  used  for  each  objective  in  the  strategic  plan.  The  report  indicates 
the  need  for  a  strong  review  process  to  ensure  individual  measures  are  relevant  and 
necessary  when  compared  to  retired  and  new  measures.  The  APR,  however,  fails  to 
integrate  or  “roll  up”  the  individual  measures  within  each  objective  to  assess  overall 
success.  Likewise,  the  individual  objectives  are  not  integrated  to  assess  the  aligned  goal. 

184  Government  Performance  and  Results  Act  (GPRA)  Modernization  Act  of  20 10,  Pub.  L.  No.  111- 
352,  124  Stat.  3866  (2011). 

185  FEMA,  National  Preparedness  Goal,  A-l. 
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In  addition,  the  graphic  display  of  relevant  trends  in  the  APR  would  provide  a 
quick  perfonnance  assessment.  In  one  specific  case,  the  stated  performance  of  the 
Compliance  rate  for  Customs-Trade  Partnership  Against  Terrorism  showed  a  steady 
negative  trend  over  a  five-year  period  and  was  subsequently  retired,  but  lacked 
explanation  for  the  measurement  change.186  Analysis  of  the  report  shows  33  percent  of 
the  total  measures  indicated  performance  targets  of  94  percent  or  greater  level  and  20  of 
the  measures  stated  performance  targets  of  100  percent.187  Even  though  the  target 
performance  level  may  be  100  percent,  it  would  be  instructive  to  understand  how  many 
resources  would  be  required  to  achieve  that  level  of  performance,  and  if  perfection  is 
necessary,  or  even  a  reasonable  expectation. 

D.  DHS  ASSESSMENT  SYSTEMS 

Congress  has  demanded  accountability  from  the  homeland  security  enterprise 
through  performance  measurement;  however,  DHS  has  struggled  to  develop  a  reliable 
measurement  system  that  could  account  for  the  homeland  security  enterprise’s  evolving 
nature.  In  order  to  comply  with  the  performance  assessment  requirements,  DHS  has 
established  numerous  systems  designed  to  measure  capabilities,  readiness,  and 
preparedness:  the  State  Preparedness  Reporting  System  (SPR),  the  National  Incident 
Management  System  Compliance  Assistance  Support  Tool  (NIMSCAST),  the  Grants 
Reporting  Tool  (GRT),  the  Logistics  Capability  Assessment  Tool  (LCAT),  the  Gap 
Analysis  Program  (GAP),  and  the  Cost-to-Capability  (C2C)  pilot  program.  FEMA  has 
yet  to  implement  the  five-step  central  repository  for  national  preparedness  data — known 
as  the  Comprehensive  Assessment  System — called  for  in  2003’s  HSPD-8,  and  later  in 
2006  with  the  PKEMRA.  The  following  sections  summarize  the  current  systems 
employed  to  address  FEMA’s  performance  assessment  requirement. 


186  DHS,  Annual  Performance  Report.  Fiscal  Years  2012-2014,  22. 

187  Department  of  Homeland  Security,  Annual  Performance  Report.  Fiscal  Years  2012-2014 
(Washington,  DC:  Department  of  Homeland  Security,  2013),  14-44. 
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1.  Capabilities  Assessment  for  Readiness 

The  Capability  Assessment  for  Readiness  (CAR)  was  the  first  attempt  to  develop 
a  nationwide  assessment  of  emergency  management  performance.  The  CAR  was  created 
in  1997  in  response  to  a  congressional  inquiry  that  called  for  “a  national  set  of  emergency 
management  perfonnance  criteria  for  FEMA  grant  recipients.”188  CAR  was  a  one-time 
assessment  of  capabilities  completed  over  a  three-month  period  through  self-assessments 
by  56  states  and  territories.  The  “CAR  lacked  controls  for  accurately  validating  self- 
reported  assessment  data,  which  limited  the  reliability — and  therefore  the  value  of — the 
data.”189 

2.  Gap  Analysis  Program 

The  GAP  began  in  2007  and  was  designed  to  assess  and  improve  the  response 
readiness  of  federal,  state  and  local  entities  by  assessing  the  gaps  in  disaster-response 
capabilities.  Initially,  FEMA  assessed  the  response  of  states  and  territories  subject  to  a 
Category  3  hurricane.  The  GAP  looked  at  seven  critical  response  missions  of  the  37 
target  capabilities,  including:  “debris  removal,  commodity  distribution,  evacuation, 
sheltering,  interim  housing,  fuel  capacity  along  evacuation  routes,  and  communications, 
but  did  not  include  comprehensive  information  for  activities  related  to  all  37  target 
capabilities.”190  In  2009  GAP  was  expanded  nationwide  to  better  assess  overall 
readiness.191 

The  GAP’s  advertised  strengths  included  a  standard  tool  to  estimate  the 
capabilities  and  resources  needed  for  an  effective  response  that  allowed  input  from  state 
and  local  emergency  managers  and  first  responders.  The  Department  of  Health  and 
Human  Services  (HHS)  assessed  medical  needs  from  data  provided  by  states  and 
territories.  Finally,  the  GAP  utilized  a  standardized  data  collection  and  analysis  tool 


188  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  108. 

189  Ibid. 

190  Ibid.,  55,  Table  2. 

191  Ibid.,  106. 
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(DCAT)  that  measured  preparedness  through  readiness  indicators  and  quantifiable 
resource  data  that  could  be  used  with  any  hazard  (not  just  hurricanes).192 

It  also  ultimately  provided  states  flexibility  to  use  scenarios  tailored  to  their 
unique  risks,  such  as  hurricanes  or  earthquakes.  Several  limitations  of  the  program, 
however,  were  also  identified.  First,  states  lacked  adequate  resources  to  input  accurate 
capability  information  into  the  GAP.  Next,  process  lacked  standardization  resulting  in  an 
unbalanced  analysis  across  the  participating  states,  which  added  uncertainty  to  the  results. 
Finally,  capability  requirements  could  not  be  determined  due  to  the  lack  of  a  national 
planning  system  and  the  inability  of  differing  levels  of  government  to  collaborate.  As  a 
result,  the  GAP  was  no  longer  considered  a  solution  to  the  national  preparedness 
assessment  challenge.193 

3.  State  Preparedness  Report 

First  developed  and  submitted  in  2008,  the  SPR  is  a  PKEMRA-mandated 
reporting  requirement.  All  56  states  and  territories  were  mandated  to  complete  the  report 
as  a  condition  to  receive  homeland  security  grant  funding.  The  SPR  assesses  current 
capability  levels,  identifies  unmet  target  capabilities,  and  assesses  resource  gaps.  States 
continue  to  submit  the  SPR  as  a  grant  requirement,194  however,  “FEMA  could  not  use 
information  in  the  first  reports  to  compare  capability  gaps  between  states  because  states 
did  not  report  information  using  common  metrics,  and  data  were  not  always  available  to 
consistently  complete  the  report.”195 

The  Federal  and  State  Preparedness  Reports  each  have  an  annual  requirement. 
The  federal  report  is  used  to  classify  the  Nation’s  level  of  preparedness  and  assesses  how 
well  federal  assistance  supports  national  preparedness  efforts  and  resources.  The  federal 


192  FEMA,  Gap  Analysis  Program  Guidance  (Washington,  DC:  Department  of  Homeland  Security, 
September  2009),  March  13,2009,  https://www.hsdl.org/?view&did=34389. 

193  GAO,  FEMA  Has  Made  Limited  Progress,  21,  http://www.gao.gov/assets/100/97160.pdf. 

194  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
Management  Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1430  (2006). 

195  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  55,  Table  2. 
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report  is  submitted  to  appropriate  congressional  committees.196  Each  report  is  required  to 
assess  state  compliance  with  national  plans  and  strategies,  identify  current  and  target 
capability  levels,  and  list  the  resource  gaps  necessary  to  meet  the  target  capabilities  based 
on  national  priorities.197 

The  SPR  is  not  considered  an  assessment;  FEMA  uses  the  data  from  the  SPRs, 
along  with  other  data  and  reports,  to  create  the  National  Preparedness  Report .198  In 
2010,  FEMA  improved  the  reporting  process  by  developing  a  web-based  survey  and 
included  standardized  measures  to  assess  progress  in  achieving  the  Target  Capabilities 
List.  Since  2010,  however,  the  Target  Capabilities  List  has  been  abandoned,  and  replaced 
by  the  National  Preparedness  Goal’s  31  Core  Capabilities,  making  comparative  analysis 
difficult.199  Additionally,  the  reports  offer  limited  data  reliability  due  to  the  non-standard 
self-reporting  process.200  FEMA  officials  say  “they  have  taken  steps  to  address  these 
limitations;  and  they  believe,  overall,  the  data  they  have  are  reliable.”201 

4.  National  Incident  Management  System  Compliance  Assistance 
Support  Tool 

The  National  Incident  Management  System  Compliance  Assistance  Support  Tool 
(NIMSCAST)  provides  a  web-based,  self-assessment  instrument  for  federal  and  SLTT 
governments  to  evaluate  and  report  their  achievement  of  National  Incident  Management 
System  (NIMS)  implementation  activities.  It  is  a  self-assessment  system  used  to  identify 
successes  and  shortfalls  in  compliance,  best  practices,  and  technical  assistance  from  the 
National  NIMS  Integration  Center.  NIMSCAST  established  baseline  measures  for 
compliance  with  NIMS  standards,  primarily  through  yes-or-no  performance-based 


196  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
Management  Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1429  (2006). 

197  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
Management  Reform  Act  of  2006),  Pub.  L.  No.  109-295,  120  Stat.  1430  (2006). 

198  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts. 

199  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  17. 

200  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  20. 

201  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  13. 
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questions.  NIMSCAST  was  nationally  deployed  and  included  users  at  all  levels  of 
government.202 

Challenges  with  the  system  include  a  lack  of  standardization  on  the  entities  or 
jurisdictions  within  each  state  that  must  report  NIMS  implementation.  Also,  because 
NIMS  is  a  self-assessment  process  the  resulting  outputs  were  deemed  unreliable.203 
NIMSCAST  was  scheduled  to  be  included  in  the  PrepCAST  system  but  was  unsupported 
due  to  a  lack  of  funding.  It  is  unclear  how  NIMSCAST  will  integrate  with  future  efforts 
to  measure  target  capabilities.204 

5.  Pilot  Capability  Assessment 

The  Pilot  Capability  Assessment’s  (PCA)  purpose  was  to  create  and  test  a 
systematic,  repeatable  assessment  methodology  that  could  be  developed  into  a 
capabilities  assessment  tool,  used  to  create  a  national  picture  of  preparedness.  Moreover, 
the  PCA  assessed  a  region’s  progress  toward  self-defined  targets  for  activities  linked  to 
the  Target  Capabilities  List.  A  web-based  version  of  the  PCA  was  piloted  in  10  states  and 
an  on-site  assessment  was  piloted  in  six  states.  However,  because  “the  PCA  was  labor- 
intensive  and  did  not  generate  meaningful  data,”205  primarily  due  to  self-reporting 
limitations  and  the  associated  data  reliability  issues,  the  project  did  not  progress  beyond 
the  initial  pilot.206 

6.  Grants  Reporting  Tool 

The  GRT  is  a  reporting  system  designed  as  a  quality  assurance  system  rather  than 
as  a  preparedness  assessment  system.  Its  goal  was  to  monitor  financial  and  programmatic 


202  FEMA,  NIMS  Implementation  Matrix  for  State  and  Territories  (October  1,  2006  -  September  30, 
2007),  4,  (Washington,  DC:  Department  of  Homeland  Security,  2006);  Federal  Emergency  Management 
Agency  (FEMA),  National  Incident  Management  Compliance  Assistance  Support  Tool  (NIMSCAST) 
(February  21,  2008),  http://www.fema.gov/nimscast 

203  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  54. 

204  Ibid.,  55,  Table  2. 

205  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  55. 

206  Ibid.,  55-56. 
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information  from  SLTT  jurisdictions  for  grants  and  cooperative  agreement  awards.  The 
information  gathered  from  the  tool  enables  FEMA  “to  evaluate  applications  and  make 
award  decisions,  monitor  ongoing  grant  performance  and  manage  the  flow  of  federal 
funds,  and  appropriately  close  out  grants  or  cooperative  agreements.”207  The  GRT 
collects  information  from  each  grant  program  and  tracks  progress  in  developing 
capabilities.208 

Since  2004,  the  GRT  has  tracked  the  grant  life  cycle  of  2 1  grant  programs  and 
collected  data  on  how  SLTT  governments  planned  to  use  grant  program  dollars  and  how 
the  money  was  actually  spent.  However,  using  this  data  to  assess  preparedness 
capabilities  will  be  difficult  because  capability  improvements  are  not  part  of  the  data 
collection  regime.  Data  reliability  is  an  issue  with  GRT  due  to  self-reporting  limitations 
of  the  web-based  program  and  the  lack  of  standards  to  mitigate  individual  interpretation 
and  provide  consistency.  GRT  reports  are  completed  on  a  biannual  basis  to  FEMA  for 
monitoring  and  oversight.209 

7.  Logistics  Capability  Assessment  Tool 

The  Logistics  Capability  Assessment  Tool  (LCAT)  was  designed  as  another  self- 
assessment  tool  to  detennine  logistical  capabilities  for  disaster  planning  and  operations 
and  the  identification  of  areas  needing  improvement.  The  tool  was  not  intended  to 
provide  an  assessment  of  collective  capabilities  since  local  differences  prevent  evaluation 
at  the  regional  or  national  level.210  LCAT’s  pilot  program  was  completed  in  FY  2010  and 
is  still  available  for  use  by  the  request  of  SLTT  jurisdiction  through  FEMA’s  regional 
technical  assistance  program. 


207  Federal  Emergency  Management  Agency  (FEMA),  FEMA ’s  Grants  Reporting  Tool  (Washington, 
DC:  Department  ofFlomeland  Security,  2009),  The  Federal  Register,  59,  216, 
http://edocket.access.gpo.gov/2009/pdf/E9-27552.pdf. 

208  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  10. 
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8.  Cost-to-Capability  Pilot  Program 

The  Cost-to-Capability  (C2C)  initiative  was  a  pilot  program  launched  in  2008  to 
help  FEMA  and  localities  better  target  and  measure  federal  grant  fund.  The  C2C  used  the 
National  Planning  Scenarios  as  a  basis  for  its  framework.  Data  was  derived  from  the 
state’s  self-assessments  and  estimates  on  current  and  future  capabilities.  Because  the 
assessment  lacked  standardization  and  reliable  metrics  reliable  comparisons  could  not  be 
made  and  proved  ineffective.211  FEMA  has  not  yet  developed  these  metrics. 

The  C2C  initiative  utilized  broad  representation  from  SLTT  government  officials 
to  develop  a  data  collection  system  intended  to  measure  the  effectiveness  of  several 
programs.212  The  initiative,  known  as  the  Reporting  Requirements  Working  Group, 
worked  with  FEMA  to  find  areas  where  current  data  collection  efforts  could  be 
combined.  In  their  final  report,  they  concluded  surveys  best  meet  stakeholder 
requirements  for  a  data  collection  tool.213 

The  C2C  program  “was  intended  to  optimize  the  impact  of  homeland  security 
grant  dollars  on  preparedness  efforts  and  measure  the  return  on  investment  grants.”214 
However,  in  July  2009,  FEMA  suspended  the  collection  of  data  from  SLTT  jurisdictions. 
As  a  result,  the  C2C  initiative  was  suspended  in  November  2009.215 

The  GAO  noted  a  limitation  of  C2C:  FEMA  did  not  require  that  grantees  utilize 
C2C,  which  limited  the  amount  of  infonnation  collected  and  resulted  in  questionable 
overall  value.216  Thus,  FEMA  could  not  “collect  comprehensive  infonnation  to  assess 
how  grant  funds  have  improved  capabilities.”217  Additionally,  C2C  lacked  a  standardized 

211  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  Exercise  and 
Assessment  Efforts,  107. 

212  U.S.  Department  of  Homeland  Security  Office  of  Inspector  General,  FEMA ’s  Preparedness  for  the 
Next  Catastrophic  Disaster  -An  Update  (OIG  10-123),  September  2010,  23, 
http://www.oig.dhs.gov/assets/Mgmt/OIG_10-123_SeplO.pdf. 

213  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  36. 

214  U.S.  Department  ofHomeland  Security  Office  of  Inspector  General,  FEMA ’s  Preparedness,  12. 

215  U.S.  Department  ofHomeland  Security  Office  of  Inspector  General,  FEMA ’s  Preparedness,  12. 
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formula  for  determining  necessary  costs,  and  users  assessed  their  needs  using  individual 
judgment.218 


9.  Comprehensive  Assessment  System 

In  December  2008,  FEMA  provided  plans  to  establish  the  Comprehensive 
Assessment  System  (CAS)  by  May  2010,  to  “function  as  a  central  repository  for  national 
preparedness  data.”  The  system  was  to  “integrate  data  from  prior  reports  and  legacy 
assessment  systems.”219  The  CAS  was  intended  to  “assess  the  nation’s  prevention 
capabilities  and  overall  preparedness,  including  operational  preparedness.”  The 
assessment  system  was  tasked  to  measure:  compliance  with  the  national  plans  and 
strategies;  capability  levels;  resource  needs;  and,  finally,  the  performance  of  training, 
exercises,  and  operations.  PKEMRA  mandated  that  specific  plans  and  strategies  within 
each  component  of  the  National  Preparedness  System  “should  be  updated  with  clear  and 
quantifiable  performance  metrics,  measures,  and  outcomes.”220 

The  CAS  (as  shown  in  Figure  1)  was  designed  to  integrate  data,  conduct  analysis, 
and  report  on  the  state  of  national  preparedness.  It  was  not  defined  as  a  new  data 
collection  tool,  IT  platform,  or  automated  calculator.  To  date,  data  requirements  and 
metrics  have  yet  to  be  defined.  Although  there  is  plenty  of  data  available  to  collect,  such 
as  the  SPR,  Grant  Reporting  Tool,  Logistics  Capabilities  Assessment  Tool,  and  data  from 
exercises  and  disaster  response  after  action  reports,  it  is  unclear  how  the  data  can  be 
integrated  and  analyzed  to  provide  meaningful  and  actionable  information  to  decision 
makers. 


218  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  22. 
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220  Department  of  Homeland  Security  Appropriations  Act  2007  (Post-Katrina  Emergency 
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Source:  GAO  analysis  of  FEMA  data. 

Figure  1.  GAO  Representation  of  the  Comprehensive  Assessment  System221 


The  primary  function  of  CAS,  as  illustrated  in  Figure  1,  is  a  framework  that 
brings  together  multiple  sources  of  data  based  on  established  metrics.  It  appears  the  CAS 
will  serve  merely  as  single  portal  to  access  the  various  systems  and  tools  already  in  place 
but  does  not  integrate  the  information  or  provide  any  additional  analysis  capability.  It 
should  be  noted  the  Gap  Analysis  Program,  Cost-to-Capability  Program,  and  the  National 
Incident  Management  Systems  Compliance  Assistance  Support  Tool  all  listed  in  Figure  1 
are  no  longer  active  programs.  Best  practices  and  lessons  learned  are  two  other  existing 
programs  not  identified  that  FEMA  is  considering  including  in  the  new  system.222 

10.  National  Preparedness  System 

The  NPS  was  a  web-based  system  that  was  field  tested  in  ten  states  through 
January  2008.  The  NPS  assessed  jurisdictions  against  the  37  TCL  target  capabilities  and 
was  intended  to  guide  users  through  a  comprehensive  planning  and  assessment  process 
that  clarified  roles,  informed  investments,  and  supported  strategic  decision-making.  One 
of  TCL’s  key  benefits  was  that  it  would  determine  capabilities  gaps,  where  they  were 
needed,  at  what  level  they  were  required  to  function,  and  what  the  current  capability 

221  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  28. 
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levels  were  in  high-risk  locations.  The  ability  to  conduct  this  type  of  analysis  could 
inform  appropriators  on  how  much  funding  should  be  allocated  by  the  federal 
government  to  SLTT  jurisdictions  and  how  long  they  should  be  funded.223  However,  “the 
NPS  was  discontinued  because  it  was  time  consuming  and  did  not  produce  meaningful 
data.  Because  it  was  only  piloted,  the  NPS  did  not  generate  meaningful  preparedness 
information  from  the  data  collected.  FEMA  spent  nearly  $15  million  on  the  system  for 
2006,  2007,  and  2008  before  it  was  discontinued.”224 

11.  The  Preparedness  Compliance  Assessment  System  Tool  (PrepCAST) 

PrepCAST  is  a  system  designed  to  help  unify  FEMA’s  information  collection 
systems.  PrepCAST  is  not  designed  to  be  an  information  collection  tool,  but  rather  a 
platform  that  houses  infonnation  collection  tools.  As  the  PrepCAST  system  is  further 
developed,  it  will  allow  users  to  access  the  various  DHS  reporting  tools  through  the 
common  portal  using  a  single  set  of  login  credentials.  Although  the  initial  version  of 
PrepCAST  contains  only  a  single  application,  the  SPR,  FEMA  plans  to  expand  the 
system  to  incorporate  additional  collections,  including  NIMSCAST,  in  the  future.225 

E.  ANALYSIS  OF  DHS  ASSESSMENT  SYSTEMS 

Since  1997,  FEMA  has  attempted  to  implement  a  comprehensive  assessment 
system  that  captures  and  reports  on  national  capabilities  and  preparedness.  As  reviewed 
in  this  chapter,  DHS  and  FEMA  have  developed  over  ten  programs  to  accomplish  this 
task  and,  as  of  this  writing,  there  is  no  specific  project  or  program  identified  to 
accomplish  this  goal. 


223  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security’  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  2013)  (statement  of 
Matt  A.  Mayer,  Visiting  Fellow,  Douglas  and  Sarah  Allison  Center  for  Foreign  Policy  Studies,  The 
Heritage  Foundation). 

224  GAO,  FEMA  Has  Made  Progress  but  Needs  to  Complete  and  Integrate  Planning,  55. 

225  PrepCAST,  PrepCAST  web  page,  accessed  October  5,  2013. 
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The  review  of  each  system  reveals  several  recurring  themes  related  to  the  reported 
challenges  and  shortcomings:  self-assessing  and  self-reporting;  lack  of  a  comprehensive 
or  all-inclusive  management  system;  no  established  metrics  or  standards;  a  labor 
intensive  process  and  a  lack  of  resources  to  manage  the  system;  and,  finally,  inconsistent, 
inaccurate,  and  unreliable  data. 

The  THIRA  and  SPR  processes  are  currently  the  two  primary  avenues  for  SLTT 
jurisdictions  to  provide  input  into  the  National  Preparedness  Report.  Although  there  is  no 
current  requirement,  the  THIRA  and  SPR  are  scalable  to  allow  jurisdictions  below  the 
state  level  to  participate.  The  state  reports  are  designed  to  track  progress  toward  meeting 
preparedness  goals.  The  reports  are  also  used  to  justify  requests  for  grant  funding  by 
identifying  capability  gaps  and  the  plan  or  strategy  to  use  grant  funds  to  build  or  develop 
the  lacking  capabilities.  The  justification  is  included  in  the  state  grant  application,  along 
with  specific  outcomes  as  a  result  of  the  grant  investment.226 

In  2010,  FEMA  responded  to  a  DHS  OIG  report  on  preparedness  that  refuted  the 
OIG  assertion  that  FEMA  has  not  substantially  improved  its  assessment  process  of  SLTT 
jurisdictions.  FEMA  argues  the  SPRs  have  “substantially  improved  their  assessments  of 
capabilities/readiness  at  the  national,  state,  and  local  levels.”227  While  the  National 
Preparedness  Report  has  emerged  as  the  preeminent  document  to  assess  national 
preparedness,  there  are  still  gaps  in  data  that  affect  the  report’s  ability  to  impact  decisions 
on  strategy  and  priorities. 

To  comply  with  the  annual  requirement  to  report  on  the  state  of  national 
preparedness  to  the  president,  FEMA  compiles  data  from  a  variety  of  sources  and  works 
in  close  coordination  with  whole  community  partners.  FEMA’s  approach  to  developing 
the  report  includes  the  following  activities: 


226  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11:  Hearing  Before  the 
Senate  Committee  on  Homeland  Security’  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  1 13th  Cong.,  (2013)  (statement  of 
Timothy  Manning,  Deputy  Administrator,  Protection  and  National  Preparedness,  FEMA,  DHS), 
http://www.hsgac.senate.gov/subcommittees/emdc/hearings/are-we-prepared-measuring-the-impact-of- 
preparedness-grants-since-9/1 1. 

227  Department  of  Homeland  Security  Office  of  Inspector  General,  FEMA  ’s  Preparedness  for  the 
Next  Catastrophic  Disaster— An  Update,  (OIG  10-123),  September  2010,  Appendix  B,  61, 
http://www.oig.dhs.gov/assets/Mgmt/OIG_10-123_SeplO.pdf. 
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•  Reviewing  Threat  and  Hazard  Identification  and  Risk  Assessments 
(THIRA)  and  2013  State  Preparedness  Report  (SPR)  submissions  from 
states  and  territories 

•  Conducting  outreach  to  preparedness-related  professional  organizations 
and  associations 

•  Conducting  research  to  update  key  findings  from  previous  National 
Preparedness  Reports  and  to  identify  new  qualitative  and  quantitative 
preparedness  data  across  all  3 1  core  capabilities  identified  in  the  Goal 

•  Evaluating  results  from  established  preparedness  programs,  including 
exercises  and  grants 

•  Soliciting  federal  departments  and  agencies  for  information  on  notable 
accomplishments  and  challenges  in  enhancing  preparedness 

•  Soliciting  updates  from  federal  partners  on  their  operational  capability  to 
meet  the  Goal  and  their  progress  in  implementing  the  National  Incident 
Management  System 

•  Partnering  with  the  U.S.  Department  of  Homeland  Security  (DHS)  Office 
of  Infrastructure  Protection  to  collect  inputs  from  federal  partners  on 
activities  undertaken  to  strengthen  infrastructure  security  and  resilience.228 

These  data  collection  sources  involve  a  broad  range  of  stakeholders  but  there  is  no 
specificity  on  the  response  rate  as  a  result  of  the  solicitation  efforts  and  outreach  or  on  the 
quality  of  the  infonnation  received.  The  stated  intent  of  the  National  Preparedness 
Report  “is  to  provide  the  nation  with  practical  insights  on  core  capabilities  that  can 
inform  decisions  about  program  priorities,  resource  allocation,  and  community 
actions.”229  The  2014  NPR  provides  seven  key  findings  as  a  result  of  the  data  collection 
and  analysis  process: 

•  New  Approach  to  Recovery:  Major  events,  such  as  Hurricane  Sandy  and 
the  severe  2012-2013  drought,  have  served  as  catalysts  for  change  in 
national  preparedness,  drawing  clearer  links  between  post-disaster 
recovery  and  pre-disaster  mitigation  activities. 

•  National  Areas  for  Sustaimnent:  Continued  progress  has  resulted  in 
several  mature  capabilities  across  multiple  mission  areas  that  require 
ongoing  sustainment  to  meet  expected  future  needs. 


228  FEMA,  National  Preparedness  Report,  1. 

229  Ibid. 
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•  National  Areas  for  Improvement:  Cyber  security,  Infrastructure  Systems, 
Health  and  Social  Services,  and  Housing  remained  national  areas  for 
improvement.  Long-term  Vulnerability  Reduction  is  a  newly  identified 
national  area  for  improvement. 

•  Emergency  Policy  and  Planning  Initiatives:  New  national  policy  and 
planning  initiatives  are  focusing  efforts  to  address  areas  for  improvement 
in  preparedness  and  national  risk  priorities. 

•  Impact  of  Budget  Uncertainties:  Budget  uncertainties  have  created 
challenges  for  preparedness  initiatives,  resulting  in  increased  emphasis  on 
preparedness  innovations  and  whole  community  engagement. 

•  Self-assessment  Results  from  States  and  Territories:  States  and  territories 
assessed  their  capabilities  similar  to  levels  reported  in  2012,  with  the 
highest  self-assessment  scores  in  the  common  core  capabilities  and  the 
Response  mission  area.  States  and  territories  also  reported  the  most  annual 
progress  in  Operational  Coordination,  Planning,  and  Intelligence  and 
Information  Sharing. 

•  Integrating  Tribal  Partners:  The  Nation  is  integrating  tribal  partners  more 
systematically  into  preparedness.  However,  challenges  remain  for  federal 
agencies  and  tribal  nations  to  increase  engagement  and  expand  training 
opportunities  on  relevant  policies.230 

These  key  findings  are  strategic  in  nature,  but  it  is  questionable  how  much  reliable 
data  is  available  to  analyze  and  compare.  This  makes  it  difficult  to  provide  reasonable 
judgments  and  decision-quality  recommendations.  For  instance,  in  the  2014  National 
Preparedness  Report  the  states  and  territories  reported  capability  levels  similar  to  the 
2013  assessment,  while  at  the  same  time  reporting  that  declines  in  state  budgets  and  high 
turnover  rates  among  staff  “contributed  to  gaps  in  28  of  the  31  core  capabilities.”231 
There  is  either  reason  to  question  the  results  of  the  individual  reports,  or  capabilities  are 
somehow  being  sustained  nationwide  with  fewer  available  resources. 

Measurement  subjectivity  is  also  evident  in  the  2012  and  2013  NPRs.  In  2012,  the 
average  preparedness  percentage  across  all  core  capabilities  was  62  percent.232  In  2013, 


230  Ibid.,  i. 

23 1  FEMA,  National  Preparedness  Report,  9. 

232  Ibid.,  ii. 
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the  percentage  fell  to  just  43  percent.233  It  is  unreasonable  to  think  that  an  investment  in 
excess  of  a  billion  dollars  into  the  national  preparedness  system  resulted  in  a  nearly  20 
percent  drop  in  capability.234  In  this  case,  subjectivity  seems  to  have  rendered  these 
reports  meaningless.  The  2014  and  2015  National  Preparedness  Reports  state  an  average 
of  40. 4235  and  42. 5236  respectively  showing  some  stability  in  the  data  but  no  trend  of  an 
increased  capability  is  evident. 

Another  discrepancy  occurs  when  the  states’  self-reported  ratings  are  compared 
with  independent  assessments  of  the  Emergency  Management  Accreditation  Program 
(EMAP).  According  to  the  March  30,  2013  NPR,  planning  was  one  of  the  10  highest 
rated  capabilities  among  the  states.237  55  percent  of  the  States  and  Territories  ranked 
themselves  4  or  5  on  the  Planning  capability  within  the  NPR.238  This  self-rated  capability 
remained  stable  at  52  percent  in  the  2014  NPR.239  However,  the  top  three  non-compliant 
standards  in  both  the  2012  and  2013  EMAP  annual  analysis  reports  were  contained 
within  the  operational  planning  section  of  the  Emergency  Management  Standard. 
Programs  undertaking  an  EMAP  assessment  were  found  to  struggle  to  document 
compliance  with  common  planning  elements  for  continuity  of  operations  and  continuity 
of  government.240 

Prior  to  the  release  of  the  first  NPR,  FEMA  officials  said  that  “their  evaluation 
efforts  were  useful  for  their  respective  purposes,  but  the  national  preparedness 

233  Ibid.,  8. 

234  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/1 1,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
Matt  A.  Mayer,  Visiting  Fellow,  Douglas  and  Sarah  Allison  Center  for  Foreign  Policy  Studies,  The 
Heritage  Foundation). 

235  FEMA,  National  Preparedness  Report,  10. 

236  Ibid.,  15. 

237  Ibid.,  8. 

.23  8lbid. 

239  Ibid.,  10,  Figure  2. 

240  Emergency  Management  Accreditation  Program,  Assessment  Analysis  Report,  January ;  - 
December  2013  (Lexington,  KY:  Emergency  Management  Accreditation  Program,  2014),  5, 
http://www.emaponline.org/index.php/root/assessor-resources/47-2013-assessment-analysis-report-l, 
Accessed  November  8,  2014. 
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capabilities  data  that  have  been  collected  to  date  through  selected  evaluation  efforts  were 
limited  by  data  reliability  and  measurement  issues  related  to  the  lack  of  standardization  in 
data  collection.”241  It  is  unclear  if  data  quality  has  improved  since  2011  to  substantially 
change  the  overall  evaluation  effort.  “Despite  the  important  role  of  individuals  and 
communities  in  preparing  for  a  disaster,  FEMA  faces  numerous  challenges  in  measuring 
the  effectiveness  of  its  efforts  to  enhance  individual  and  community  preparedness.”242 

The  second  key  component  of  FEMA’s  current  assessment  methodology  is  the 
THIRA.  As  part  of  the  process,  state  and  local  governments  are  to  develop  individual 
capability  requirements.  The  collective  state  THIRA  reports  are  intended  to  provide 
FEMA  with  sufficient  data  to  analyze  and  set  national  funding  priorities.243  Completion 
of  the  THIRA  is  a  requirement  for  all  state  and  local  governments  receiving  homeland 
security  funding.244 


241  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  13. 

242  Department  of  Homeland  Security  Office  of  Inspector  General,  FEMA  ’s  Preparedness  for  the 
Next  Catastrophic  Disaster  -An  Update  (OIG  10-123),  September  2010,  Appendix  B,  14, 
http://www.oig.dhs.gov/assets/Mgmt/OIG_10-123_SeplO.pdf. 

243  Maurer,  National  Preparedness,  FEMA  Has  Made  Progress,  12. 

244  Ibid.  (Highlights). 
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IV.  CONCEPTUAL  MODEL 


The  development  of  the  model  for  perfonnance  management  in  assessing  national 
preparedness  is  guided  by  the  historical  limitations  and  system  requirements  as 
established  by  federal  law.  This  chapter  begins  by  reviewing  and  analyzing  the  historical 
challenges  and  limitations  of  performance  management  system  development.  It  then 
explores  several  concepts  that  would  mitigate  these  challenges  and  facilitate  a  process  to 
meet  the  GPRAMA’s  intent. 

A.  HISTORICAL  CHALLENGES  AND  GAPS  IN  PERFORMANCE 

ASSESSMENT 

1.  Enterprise  Complexity 

As  the  fourth  largest  federal  agency  (following  the  Departments  of  Veterans 
Affairs,  Anny,  and  Navy)  and  the  sixth  largest  department  for  federal  discretionary 
budget  authorization,  DHS’s  240,000  personnel  and  16  distinct  congressionally  funded 
organizational  components  perform  a  daunting  array  of  missions.245  Each  functional  area 
maintains  its  own  level  of  complexity  and  unique  challenges  with  evolving  threats  and 
hazards,  resource  constraints,  and  high  expectations. 

In  addition  to  the  challenges  of  the  Department’s  sheer  size  and  scope  of  mission, 
DHS  is  still  immature;  it  continues  to  evolve  as  new  or  unforeseen  consequences  confront 
our  understanding  of  how  to  best  protect,  mitigate,  respond  to,  recover  from,  and  prepare 
for  future  events.  Some  of  the  most  dramatic  organizational  changes  in  DHS  occurred  in 
2006,  due  to  post-Hurricane  Katrina  and  Hurricane  Rita  legislative  action.  Significant 
changes  also  occurred  (primarily  in  policy  and  programmatic  areas)  in  2013  following 
Super  Storm  Sandy.  Coordinating  and  implementing  these  transformational  changes 


245  Office  of  Personnel  Management,  ‘‘‘Frequently  Asked  Questions,” 
http://www.0pm.g0v/F  AQs/QA.aspx?fid=56538f91-625a-4333-84ba-28b3574b7942&pid=013be2c9-8ae5- 
455d-889c-6e74063441ba,  answer  to  the  question  “What  are  the  largest  federal  agencies,”  As  of  September 
2013;  Office  of  Management  and  Budget,  FY  2015  Budget  of  the  U.S.  Government,  203,  Table  S-l  1, 
http://www.whitehouse.gov/omb/budget/Overview;  Department  of  Flomeland  Security,  Agency  Financial 
Report,  Fiscal  Year  2014  (Washington,  DC:  Department  of  Flomeland  Security,  2014),  6-8, 
http://www.dhs.gov/sites/default/files/publications/OCFO/dhs_agency_fmancial_report_fy2014_vol2.pdf. 
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consumes  significant  time  and  resources  on  top  of  the  daily  activities  needed  to  meet 
strategic  goals  and  objectives. 

Three  factors  that  intensity  the  complexity  of  DHS’s  mission  are: 

1 .  The  threats  are  poorly  understood.  From  homegrown,  violent  extremism  to 
nuclear  or  cyber  terrorism  and  climate  change,  the  variety  and  ambiguity 
of  threats  to  consider  requires  rigorous  analysis  and  prioritization. 

2.  The  consequences  of  these  threats  and  the  means  to  mitigate  or  prevent 
them  affect  the  nation  in  many  ways.  How  can  our  communities,  critical 
infrastructure,  and  modes  of  communication  be  protected  without 
affecting  the  quality  of  the  system,  and  what  are  the  factors  that  set 
priorities? 

3.  Solutions  require  multiple  capabilities  and,  thus,  integrated  coordination 
and  planning  among  multiple  DHS  component  agencies  and  stakeholders. 
DHS  must  decide  how  to  allocate  resources  across  the  homeland  security 
enterprise,  or  whole  community,  to  most  effectively  solve  problems  of 
national  concern.246 

The  concept  of  a  “whole  community”  approach  emerged  as  a  key  factor  in 
national  preparedness  in  2008,  when  PPD-8  was  signed  into  law.  PPD-8  directed  an 
“integrated,  all-of-Nation,  capabilities-based  approach  to  preparedness.”247  The  National 
Preparedness  Goal  defined  whole  community  as  “a  focus  on  enabling  the  participation  in 
national  preparedness  activities  of  a  wider  range  of  players  from  the  private  and  nonprofit 
sectors,  including  nongovernmental  organizations  and  the  general  public,  in  conjunction 
with  the  participation  of  federal,  state,  and  local  governmental  partners  in  order  to  foster 
better  coordination  and  working  relationships.”248  The  tenn  is  sometimes  used 
interchangeably  with  “all-of-nation.”  See  Figure  2  for  a  visual  representation  of  the  term. 


246  Wills,  Strengthening  Strategic  Planning,  4.  h. 

247  Obama,  Presidential  Policy  Directive  8. 

248  FEMA,  National  Preparedness  Goal,  A-2. 
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27. 9M  Private  Enterprises 


316M+  Citizens 


Figure  2.  Whole  of  Community 


More  recent  DHS  strategic  documents  include  whole  community  as  a  core 
concept.  The  Quadrennial  Homeland  Security  Review  (QHSR)  included  the  term  as  an 
“area  of  priority”  for  development,  and  it  is  mentioned  13  times  throughout  the  report. 
Likewise,  the  latest  DHS  Strategic  Plan  mentions  whole  community  six  times.  Three  core 
principles  are  used  to  define  the  whole  community  approach:  (1)  “understanding  and 
meeting  the  actual  needs  of  the  whole  community;  (2)  engaging  and  empowering  all  parts 
of  the  community;  and  (3)  strengthening  what  works  well  in  communities  on  a  daily 
basis.”249  The  whole  community  includes  the  following  subgroups: 

•  Individuals  and  families,  including  those  with  access  and  functional  needs 

•  Businesses 

•  Faith-based  and  community  organizations 

•  Nonprofit  groups 

•  Schools  and  academia 

•  Media  outlets 


249  DHS,  The  2014  Quadrennial  Homeland  Security  Review,  73. 


•  All  levels  of  government,  including  federal,  state,  local,  tribal,  and 
territorial  partners250 

DHS  relies  heavily  on  the  states  and  territories  for  whole  community  engagement; 
the  NPR  describes  the  output  from  the  integrated  self-assessment  process  to  be  based  on 
“multi-disciplinary,  statewide  efforts  to  evaluate  risk  and  assess  preparedness,  engaging 
representatives  from  law  enforcement;  fire  service  agencies;  public  health  and  medical 
systems,  including  emergency  medical  services,  hospitals,  and  healthcare  organizations; 
and  non-governmental  organizations.”251 

A  GAO  study  found  that  coordination  among  regional  jurisdictions  proved 
difficult  due  to  competing  priorities  and  general  autonomy.252  Overcoming  these 
challenges  will  require  homeland  security  managers  to  take  states’  and  municipalities’ 
many  different  operational  structures  and  civic  traditions  into  account.253  This  is  an 
enormous  challenge  that  requires  strong  leadership,  understanding,  and  commitment  at 
every  level.  Leadership  is  perhaps  the  greatest  influence  on  the  process  of  perfonnance 
management,  and  therefore  its  success.  With  a  population  as  large  and  diverse  as  the 
whole  community,  unity  of  effort  will  be  difficult  to  achieve.  It  is  essential  for  leadership 
to  prioritize  perfonnance  management  if  the  whole  community  is  to  yield  reliable 
results.254  The  key  to  incremental  progress  in  this  area  will  be  a  focused  and  cohesive 
effort  that  can  begin  breaking  down  the  national  preparedness  enterprise’s  complexities 
into  manageable  and  prioritized  segments.  Then,  the  process  of  identifying  and  collecting 
meaningful  data  can  begin. 


250  Ibid.,  74. 

251  FEMA,  National  Preparedness  Report,  1. 

252  GAO,  Urban  Area  Security  Initiative :  FEMA  Lacks  Measures  to  Assess  How  Regional 
Collaboration  Efforts  Build  Preparedness  Capabilities  (GAO  09-651)  (Washington,  DC:  U.S.  Government 
Accountability  Office,  2009),  12.  http://www.gao.gov/new.items/d09651.pdf. 

253  Ibid. 

254  Frank  Camm  et  at.  Charting  the  Course  for  a  New  Air  Force  Inspection  System,  (RAND 
Document  No.  TR-1291-AF)  (RAND  Corporation  Technical  Report  Series,  2013),  42-43, 
http://www.rand.org/content/dam/rand/pubs/technical_reports/TR1200/TR129 1/RAND_TR129 1  .pdf. 
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2. 


Self-Assessments 


There  are  several  means  currently  available  for  conducting  assessments  on  SLTT 
preparedness  programs.  These  assessments  often  have  different  goals  and  outcomes  and 
are  referred  to  under  different  names  (such  as  audits,  technical  assistance  visits, 
monitoring  visits,  or  evaluations).  Some  are  mandated  under  federal  law,  such  as  the 
required  grant  audits,  and  others  are  imposed  by  the  federal  department  or  program 
manager.  Others,  still,  are  voluntary,  such  EMAP.  While  technological  tools  and  systems 
offer  a  virtual  means  to  provide  limited  monitoring  of  programs  and  project  perfonnance, 
most  assessments  include  a  face-to-face  component  or  site  visit.  These  visits  are  referred 
to  as  external  assessments.  The  other  means  of  assessing  performance  is  through  an 
internal  process,  or  self-assessment. 

The  NPR  is  required  annually  by  PPD-8  and  summarizes  the  progress  made  in 
achieving  the  targets  established  in  the  31  core  capabilities.  Some  of  the  report’s  most 
critical  inputs  include  information  from  the  states  and  territories,  described  as  “an 
integrated  self-assessment  process.”  The  NPR  inputs  come  from  the  THIRA,  SPR 
submissions,  and  grant  reporting  updates.  All  of  this  input  is  completed  by  the  individual 
jurisdictions  on  a  self-assessment  and  reporting  basis.255 

The  self-assessment  process  is  one  of  the  fundamental  causes  of  a  system  that 
provides  inconsistent,  inaccurate,  and  unreliable  data.  And  the  resulting  measurement 
subjectivity  is  one  of  the  critical  limitations  of  the  current  reporting  system.  For  example, 
“because  states  develop  their  own  capability  requirements  and  use  individual  judgment 
rather  than  a  quantitative  standard  to  assess  preparedness  capabilities,  it  is  difficult  to 
identify  differences  and  compare  capability  levels  across  states.”256  The  policy  of 
allowing  individual  data  collection  decisions  creates  non-standard  data,  variation  among 
states  in  reporting  structures,  and  subjective  interpretation  of  the  collected  data.  These 
limitations  will  ultimately  prevent  DHS  from  comparing  metrics  across  all  levels  of 


255  FEMA,  National  Preparedness  Report,  1. 
.256  Maurer,  National  Preparedness,  12. 
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government.257  In  addition  to  the  general  reporting  variations,  there  may  also  be  undue 
influence,  either  intentionally  or  subconsciously,  on  how  the  data  is  presented.  These 
influences  are  caused  by  the  consequences  or  rewards  of  a  department  admitting 
mistakes,  programmatic  shortfalls,  or  areas  of  non-compliance.258  One  of  the  most 
important  criteria  for  preparedness  metrics  is  that  they  should  avoid  subjective 
assessments  (particularly  self-assessments)  when  possible.259 

While  much  of  the  blame  for  unreliable  and  invalid  data  stems  from  the  self- 
assessment  process,  it  also  has  known  benefits  when  incorporated  into  the  overall 
performance  management  system.  For  example,  the  U.S.  Air  Force  (USAF)  views  the 
self-assessment  as  a  foundational  element  of  their  overall  inspection  system,  calling  it 
instrumental  in  “promoting  responsibility  and  accountability  within  the  unit.”260  One 
benefit  of  the  self-assessment  process  is  that  it  develops  a  culture  that  self-identifies 
deficiencies  and  fosters  continuous  improvement.  Additionally,  it  develops  trust  among 
all  of  the  stakeholders  and  a  resulting  reduced  reliance  on  external  assessments.261 

The  key  to  effectively  using  self-assessment  in  the  perfonnance  management 
system  is  to  complement  it  with  an  independent  and  external  validation  of  the  internal 
process.  The  USAF  has  one  such  rigorous  external  inspection  program  to  follow 
individual  self-assessments.  Two  other  programs  that  utilize  the  combination  of  internal 
and  external  assessments  are  the  Baldrige  Performance  Excellence  Program  and  the 
Emergency  Management  Accreditation  Program.  Each  of  these  programs  are  discussed 
further  in  section  D  of  this  chapter. 

3.  Expertise 

One  challenge  some  historical  assessment  systems  faced  was  a  lack  of  resources 
to  properly  manage  system  inputs,  or  process  that  was  too  time  consuming.  At  the  root  of 

257  Ibid. 

258  Camm  et  at,  Charting  the  Course,  42-43. 

259  Harrison,  “Rethinking  Readiness,”  52. 

260  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  Air  Force  Publication  90-201 
(Washington,  DC:  Secretary  of  Air  Force  (IG),  2013),  19. 

261  Ibid.,  31. 
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these  two  symptoms  is  a  dearth  of  dedicated  staff  that  possesses  the  knowledge,  skills, 
and  expertise  to  effectively  work  within  the  perfonnance  management  system.  Unless 
there  are  staff  members  dedicated  to  the  performance  improvement  process,  measuring 
preparedness  can  require  trade-offs  between  the  time  staff  members  spend  on 
assessments  and  the  time  allocated  to  other  demands.262  States  generally  do  not  have 
readily  available  staff  who  maintain  the  knowledge,  skills,  and  abilities  to  run  a 
performance  management  system,  nor  do  they  have  the  resources  to  support  a  dedicated 
assessment  position.263  A  recent  RAND  study  on  the  quality  of  self-inspection  programs 
revealed  staff  may  be  well  qualified  to  perfonn  their  primary  duty  assignments  but 
unqualified  to  serve  in  a  performance  management  role;  this  is  a  potential  root  cause  for 
unreliable  reporting.264  FEMA  officials,  however,  stated  that  “the  cost  of  using  federal 
employees  or  contractors  to  collect  and  validate  preparedness  data  at  the  state  and  local 
levels  would  be  prohibitive.”265 

The  Department  of  Defense  (DOD)  addressed  the  staffing  challenges  by  levying  a 
requirement  on  all  services  to  train  and  certify  at  least  five  percent  of  an  organization’s 
staff  as  Lean  Six  Sigma  Green  Belts  (the  lowest  level  of  staff  certification),  and  one 
percent  as  Lean  Six  Sigma  Black  Belts  (the  highest  level  of  staff  certification).266  The 
distinction  of  “onboard  population”  is  significant  in  that  no  additional  positions  were 
authorized  for  the  new  performance  management  requirement.  The  intended  goal  was  to 
add  a  perfonnance  management  skill  set  to  every  unit  and  drive  improvement  from 
within.  It  is  not  clear  if  the  services  were  able  to  attain  this  goal,  although  USAF 


262  CBO,  Linking  the  Readiness  of  the  Armed  Forces  to  DOD ’s  Operation  and  Maintenance 
Spending,  (Washington,  DC:  Congressional  Budget  Office,  2011),  6, 
http://www.cbo.gov/sites/default/files/04-25-readiness.pdf. 

263  Yee  San  Su,  “On  the  Challenge  of  Assessing  National  Preparedness,”  Homeland  Security  Library, 
accessed  Aug.  10,  2015.  https://www.hsdl.org/?view&did=696054. 

264  Camm  et  at,  Charting  the  Course,  42-43. 

265  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  13. 

266  Department  of  Defense,  Implementation  and  Management  of  the  DOD- Wide  Continuous  Process 
Improvement/Lean  Six  Sigma  (CPI/LSS)  Program.  DOD  Instruction  5010.43.  Enclosure  2,  2.  a.,  2,b. 
(Washington,  DC:  Department  of  Defense,  July  17,  2009). 
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personnel  recommended  that,  in  order  to  be  fully  effective,  performance  management 
staff  must  be  added  to  the  resource  requirement  list  and  accordingly  funded.267 

4.  Value  of  Effort 

Value  of  effort  can  also  be  viewed  in  terms  of  return  on  investment.  In  the  context 
of  time  and  resources  used  to  design,  collect,  and  analyze  performance  data,  the  actual 
value  may  be  difficult  to  assess.  In  tenns  of  the  allocation  of  federal  grant  dollars, 
however,  the  value  may  only  be  evident  (at  least  from  the  grantee’s  perspective)  in  terms 
of  the  rewards.  For  example,  consider  the  three  criteria  in  determining  funding 
authorizations  for  states  and  territories  (legislative  minimum  amounts,  anticipated 
effectiveness,  and  risk  methodology).  For  the  25  states  that  receive  only  the  minimum 
amount — which  essentially  eliminates  the  risk  and  effectiveness  portions  of  the 
equation — would  the  allocation  decision  change  if  one  of  those  states  put  the  time,  effort 
and  resources  into  a  compelling  effectiveness  context?268  Full  transparency  of  the 
decision  process  is  extremely  important  to  ensure  the  proper  organizational  behavior  is 
reinforced  and  encouraged.  In  addition,  standardization  among  all  the  grant  recipients 
will  be  necessary  to  level  the  playing  field  and  prevent  the  jurisdictions  with  more 
available  resources  to  always  “one-up”  those  with  less. 

One  example  of  positive  results  yielded  from  investment  in  a  performance 
management  system  occurred  within  the  Federal  Aviation  Administration.  The 
Administration’s  voluntary  reporting  programs  were  implemented  without  a  significant 
budget  increase  due  to  program  perfonnance  improvements  that,  over  time,  offset  the 
imposed  costs.  Although  no  formal  quantitative  analysis  of  the  voluntary  reporting 
program’s  effectiveness  was  available,  there  was  unanimous  agreement  that  the  voluntary 
reporting  programs  had  been  “well  worth  the  effort,”  and  that  “the  payback  was  very 
high.”269  This  consensus  was  based  on  the  outcomes  of  data  reported,  which  included 
significant  numbers  of  safety  issues  and  resulted  in  corrective  actions.  There  is  also  a 

267  Camm  et  at,  Charting  the  Course,  46. 

268  FEMA,  Grant  Programs  Directorate  Information  Bulletin  #398a  (Washington,  DC:  Department 
ofHomeland  Security,  2014),  5,  http://www.fema.gov/media-library/assets/documents/97599. 

269  Camm  et  at,  Charting  the  Course,  52. 
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consensus  that  the  oversight  and  quality  assurance  programs  in  place  prior  to  the 
voluntary  programs  would  not  have  identified  most  of  the  safety  issues,  which  means 
proactive  corrective  actions  would  not  have  been  possible.270 

5.  Benchmarks  and  Standards 

A  significant  impediment  to  collecting  consistent,  reliable,  and  accurate 
preparedness  measures  is  the  lack  of  process  standards.  A  typical  approach  to  measuring 
performance  is  counting  resources  or  activities — such  as  how  many  exercises  are 
conducted  annually,  or  how  many  level-4  protective  suits  are  on  hand.  Although  easy  to 
conduct,  this  approach  is  risky  because  it  does  not  focus  on  the  actual  quality  or 
effectiveness  of  the  capability  that  will  have  greater  impact  on  the  outcome  of  a  response 
operation.  For  example,  two  jurisdictions  may  each  have  a  current  preparedness, 
mitigation,  or  other  relevant  plan,  but  one  jurisdiction  effectively  collaborated  with 
relevant  stakeholders  and  completed  a  plan  that  is  a  useful  reference  in  the  conduct  of 
operations.  In  contrast,  the  other  jurisdiction  produced  a  plan  merely  to  satisfy  the  federal 
grant  requirement  and  is  not  considered  a  useful  reference  for  training,  exercises  or 
operations.  Two  common  mistakes  leaders  make  are  either  to  capture  strategy  session 
decisions  in  a  glossy  pamphlet  that  is  never  used,  or  to  not  capture  the  ideas,  assuming 
the  participants  will  take  the  appropriate  action  on  their  own.271  Another  example  is  the 
significant  difference  between  counting  bodies  available  to  respond  to  specific  events  and 
ensuring  the  staff  members  are  properly  trained,  with  recent  experience  to  keep  their 
skills  sharp.  A  standardized  approach  to  these  activities  will  help  ensure  agency  processes 
and  data  collection  efforts  are  reliable. 

Benchmarking  is  described  as  “the  process  of  identifying,  learning,  adapting  to, 
and  measuring  outstanding  practices  and  processes  from  any  organization  to  improve 
performance.”272  In  short,  it  is  a  tool  used  to  seek  out  and  identify  standards.  The 

270  Ibid. 

271  Patrick  M.  Lencioni,  The  Advantage:  Why  Organizational  Health  Trumps  Everything  Else  in 
Business,  (San  Francisco,  CA:  Jossey-Bass,  2012),  134. 

272  “Benchmarking,  Best  Practices,  and  Performance  Measures  for  Public  Entity  Risk  Management- 
Guidelines,”  Public  Entity  Risk  Institute,  November  1999,  1, 
http://www.riskinstitute.org/peri/images/file/Benchmarking_Guidelines.pdf. 
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resulting  standards  could  be  considered  either  a  baseline  of  performance,  due  to 
regulations  or  other  guidance,  or  the  gold  standard  for  the  industry,  which  highlights 
ultimate  goals  for  perfonnance.  A  benchmark,  then,  is  “a  standard  or  point  of  reference 
used  to  measure  or  judge  quality  or  value.”273  The  focus  of  the  benchmarking  process 
should  be  on  supporting  the  organization’s  strategic  objectives  and  providing  challenging 
but  attainable  goals.274  Benchmarking  can  be  conducted  on  internal  elements  of  the 
organization  across  the  homeland  security  enterprise  or  other  government  agencies, 
among  private  sector  entities,  or  within  academia  or  national  foundations.  Ultimately, 
finding  the  best  benchmark  may  require  a  review  of  several  candidates  with  similar 
processes,  and  could  evolve  and  change  over  time.275 

Benchmarks  should  be  managed  by  DHS  to  establish  priorities  and  ensure 
standardization  across  the  enterprise.  Several  opportunities  exist  to  collect  and  establish 
common  standards.  Perhaps  the  most  comprehensive  established  standard  on  the  specific 
activities  of  the  homeland  security  mission  reside  within  the  Emergency  Management 
Standard. 

The  Emergency  Management  Standard  was  established  in  compliance  with  the 
American  National  Standards  Institute  requirements  and  policies.  The  program  is 
“intended  to  indicate  the  essential  components  of  a  quality  emergency  management 
program.  State  and  local  emergency  managers  created  the  standards;  they  describe  what  a 
program  should  accomplish,  but  not  necessarily  how  compliance  should  be  achieved. 
This  provides  flexibility  to  accommodate  the  variety  of  sizes  and  complexity  found  in 
state  and  local  emergency  management  programs  in  the  United  States.  Full  compliance 
with  all  the  standards  is  required  for  full  accreditation.”276  Currently,  jurisdictions 
volunteer  to  enter  the  accreditation  process  and  fund  the  external  team  that  conducts  the 

273  Ibid. 

274  Gregory  H.  Watson,  The  Benchmarking  Workbook  (Portland,  OR:  Productivity  Press,  1992)  5, 
http://www.uspto.gov/web/offices/com/oqm-old/benchmarking_workbook_9_00.pdf. 

275  Colorado  Chapter  of  the  Public  Risk  Management  Association,  Benchmarking,  Best  Practices,  and 
Performance  Measures  for  Public  Entity  Risk  Management-  Guidelines  (Fairfax,  VA:  Public  Entity  Risk 
Institute,  1999),  1,  http://www.riskinstitute.org/peri/images/file/Benchmarking_Guidelines.pdf. 

276  Emergency  Management  Accreditation  Program,  Candidate’s  Guide  to  Accreditation,  (Lexington, 
KY:  Emergency  Management  Accreditation  Program,  2011),  1. 


76 


assessment.  The  Emergency  Management  Standard  consists  of  64  standards  within  16 
broader  categories: 

•  Program  Management 

•  Administration  and  Finance 

•  Laws  and  Authorities 

•  Hazard  Identification,  Risk  Assessment  and  Consequence  Analysis 

•  Hazard  Mitigation 

•  Prevention 

•  Operational  Planning 

•  Incident  Management 

•  Resource  Management  and  Logistics 

•  Mutual  Aid 

•  Communications  and  Warning 

•  Operations  and  Procedures 

•  Facilities 

•  Training 

•  Exercises,  Evaluations,  and  Corrective  Action 

•  Crisis  Communications,  Public  Education,  and  Information277 

Another  avenue  to  identify  viable  benchmarks  and  establish  enterprise  standards 
is  the  audit,  inspection,  and  program  monitoring  functions  that  currently  exist  in  DHS. 
The  DHS  OIG  has  a  unique  opportunity  to  view  and  analyze  a  substantial  sample 
population  every  year,  and  the  entire  enterprise  over  a  multiple-year  period.  The  ability  to 
conduct  thorough  programmatic  reviews  of  a  variety  of  organizations  conducting 
identical  missions  with  similar  processes  allows  the  OIG  to  assess  and  identify  the  best 
performers  across  the  SLTT  community  and  highlight  those  techniques  and  procedures 


277  “2013  Emergency  Management  Standard,”  EMPA,  accessed  Aug.  8,  2015. 
http://www.emaponline.org/index.php/root/for-programs/23-2013-emergency-management-standard. 


77 


that  could  benefit  others.  The  USAF  conducts  a  similar  program  with  their  OIG,  in  which 
they  disseminate  information,  innovative  ideas,  and  lessons  learned  as  a  result  of 
inspections  and  changes  to  inspection  policy.278 

Other  potential  areas  to  review  and  identify  standards  within  the  homeland 
security  enterprise  include: 

•  The  National  Institute  of  Standards  and  Technology  (NIST) 
(http://www.nist.gov/).  The  NIST  standards  are  often  technical  in  nature, 
and  there  are  many  areas  of  research  within  NIST  that  directly  support  the 
homeland  security  mission. 

•  The  Baldrige  Program  (http://www.nist.gov/baldrige/).  Within  NIST,  the 
Baldrige  Program  is  “a  public-private  partnership  that  is  committed  to 
improving  competitiveness  and  performance  of  U.S.  organizations”279 
through  education  and  assessment  tools  and  criteria. 

•  The  International  Organization  for  Standardization  (ISO) 

(http://www.iso.org/):  The  ISO  has  over  19,000  standards  publications 
involving  technical  specifications,  product  standards,  process  standards, 
and  regulatory  requirements  for  products  and  associated  processes.280 

International  organizations  perfonning  the  same  mission  are  also  ideal  templates 
for  identifying  standards.  Countries  with  similar  strategic  missions  and  advanced 
homeland  security  functions  include  Great  Britain,  Israel,  and  Australia,  among  others. 

6.  Standardized  Data 

The  Goal  asserts  that  “all  levels  of  government  and  the  whole  community  should 
present  and  assess  risk  in  a  similar  manner  in  order  to  provide  a  common  understanding 
of  the  threats  and  hazards  confronting  our  nation.”281  One  of  the  key  challenges  in  using 
state  and  local  data  to  provide  a  national  preparedness  assessment  is  the  jurisdictions’ 
differences  in  tenns  of  capabilities,  culture,  threats  and  hazards,  and  population  densities, 


278  Secretary  of  Air  Force  (IG ),  Air  Force  Inspection  System,  Air  Force  Publication  90-201 
(Washington,  DC:  Secretary  of  Air  Force  (IG),  2013),  10. 

279  “Baldrige  Performance  Excellence  Program  Mission  Statement,’’  NIST,  accessed  Aug.  8,  2015. 
http://www.nist.gov/baldrige/about/index.cfm. 

280  International  Organization  for  Standardization,  International  Standard  9000;  Quality  Management 
Systems — Fundamentals  and  Vocabulary-Third  Edition,  September  15,  2005,  2. 

281  FEMA,  National  Preparedness  Goal,  3. 
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to  name  a  few.  Given  their  uniqueness,  is  there  a  way  to  assess  the  individual  states  and 
jurisdictions  while  maintaining  the  capability  to  combine  the  data  into  a  cohesive  and 
accurate  national  picture  of  readiness?  “While  we  have  a  natural  predilection  for  a  picture 
of  national  preparedness  derived  from  a  commonly  applied  set  of  rules,  we  must  come  to 
terms  with  the  dissonances  found  in  each  state.  This  will  require  a  mastery  of 
fundamental  principles,  an  awareness  of  cognitive  pitfalls,  flexibility  in  how  we  structure 
our  objectives,  a  willingness  to  assimilate  and  adapt,  and  a  mechanism  by  which  we  can 
accelerate  the  pace  of  development.”282 

Data  can  be  difficult  to  collect  and  is  problematic  to  correlate  when  there  are 
differing  reporting  standards.  One  potential  source  of  standardized  and  reliable  data  can 
come  from  DHS  OIG  reports.  These  audits  are  used  to  ensure  that  states  comply  with 
applicable  laws  and  regulations  in  the  execution  of  their  grant  award.  The  reports  can  be 
considered  a  reliable  source  for  data  because  the  OIG  uses  “generally  accepted 
government  auditing  standards”  (GAGAS)  that  provide  a  consistent  and  high-quality 
framework  for  conducting  audits.283 

In  addition  to  the  financial  and  legal  aspects  of  grant  management,  the  OIG  also 
reviews  some  management,  programmatic,  and  administrative  aspects  of  homeland 
security.  Recent  findings  show  “challenges  related  to  the  states’  homeland  security 
strategies,  obligation  of  grants,  reimbursement  to  subgrantees  for  expenditures, 
monitoring  of  subgrantees ’  perfonnance  and  financial  management,  procurement,  and 
property  management.”284  These  findings  have  potential  to  provide  valuable 
contributions  to  perfonnance  measurements. 

OIG  reports  show  states  did  not  always  obligate  Homeland  Security  Grant 
Program  (HSGP)  grants  to  subgrantees  in  a  timely  manner.  This  practice  has  the  potential 


282  Yee  San  Su,  “On  the  Challenge  of  Assessing  National  Preparedness.” 

283  Government  Accountability  Office  (GAO),  Government  Auditing  Standards  2011  Revision, 
(GAO-12-331G),  (Washington,  DC:  U.S.  Government  Accountability  Office,  2011),  5, 
http://www.gao.gov/products/GAO-12-331G. 

284  Department  of  Homeland  Security  Office  of  Inspector  General,  Countering  Terrorism;  Semi- 
Annual  Report  to  the  Congress  (Washington,  DC:  DHS,  April  1,  2014 — September  30,  2014),  14-16, 
https://www.oig.dhs.gov/assets/SAR/OIG_SAR_Aprl4_Sepl4.pdf. 
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to  increase  administrative  costs  and  hinder  the  subgrantees’  ability  to  accomplish  their 
designated  capability  development  activities  or  complete  projects.  Additionally,  many 
states  did  not  provide  adequate  oversight  of  subgrantee  performance  in  execution  of  their 
allocated  grant  funds.  This  shortfall  limits  the  understanding  of  the  lower  jurisdiction’s 
progress  in  developing  or  achieving  essential  capabilities.  In  addition,  states  often  did  not 
ensure  an  alignment  of  requested  funds  to  identified  threats  and  vulnerabilities  potentially 
resulting  in  poor  management  of  scarce  resources.  Weaknesses  in  property  management 
were  also  found  to  negatively  affect  the  state’s  ability  to  make  informed  decisions  on 
eliminating  capability  gaps.285 

As  an  illustration,  the  chart  in  Figure  3  shows  the  audit  results  for  state  and 
territorial  jurisdictions  during  FY  2014.  In  all,  there  were  14  total  audits  with  only  one 
jurisdiction  reporting  zero  recommendations  for  corrective  actions.286  The  data,  if  only  at 
a  high  level,  clearly  shows  several  areas  for  concern  and  some  potential  focus  areas  for 
the  entire  enterprise. 


285  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security’  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (statement  of 
Anne  L.  Richards,  Assistant  Inspector  General  for  Audits,  DHS). 

286  Thirteen  states  were  evaluated  in  their  management  of  homeland  security  grant  programs  in  FY14. 
http://www.oig.dhs.gov/index.php?option=com_content&view=article&id=210&Itemid=198. 
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FY  14  Homeland  Security  Grant  Award  Audit  Results 


Result  Cateqorv 

# 

Reported 

Obligation  of  Grant  Funds  to  Subgrantees 

13 

Su  b  g  ra  ntee  M  on  i  torin  g/Oversig  ht 

11 

Property  Management  and  Accountability 

9 

Measuring  Preparedness  Improvements 

7 

Homeland  Security  Strategies 

6 

Procurements 

5 

Personnel  Time  Charges 

4 

Fusion  Center  Operations 

2 

Review  and  Approval  of  Grant  Expenditures  (internal 
controls) 

2 

Financial  Reporting  Accuracy 

2 

Sustainment  Capabilities 

2 

Single  Audit  Services 

1 

Preparedness  System 

1 

Training  and  Exercise  Activities 

1 

Policies  and  Procedures 

1 

Equipment  Usage 

1 

Total  Assessments  14 

Figure  3.  DHS  OIG  Homeland  Security  Grant  Award  Results287 


In  addition  to  identifying  capability  gaps  and  effectively  building  toward  closing 
those  gaps,  the  DHS  OIG  has  also  identified  programmatic  shortfalls  with  state 
preparedness  programs.  One  common  deficiency  involves  the  process  of  developing  state 
homeland  security  strategies.  The  OIG  found  many  state  strategies  lack  specific  goals  and 
objectives  and  many  others  were  outdated.  Other  states  did  not  have  an  updated  strategy 
that  reflected  current  priorities  and  threats  which  can  negatively  impact  budget 
decisions.288 

In  their  latest  annual  analysis  report,  the  EMAP  organization  outlined  three  areas 
in  which  state,  local,  and  now  private  entities  have  the  greatest  challenges  for  non- 


287  Derived  from  individual  state  and  territorial  audit  reports  taken  from  the  DHS  OIG  website, 
http://www.oig.dhs.gov/index.php?option=com_content&view=article&id=210&Itemid=199) 

288  Are  We  Prepared?  Measuring  the  Impact  of  Preparedness  Grants  Since  9/11,  Hearing  Before  the 
Senate  Committee  on  Homeland  Security  and  Governmental  Affairs,  Subcommittee  on  Emergency 
Management,  Intergovernmental  Relations  and  the  District  of  Columbia,  113th  Cong.,  (2013)  (Statement 
of  Anne  L.  Richards,  Assistant  Inspector  General  for  Audits,  DHS). 
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compliance.  All  three  areas  fall  within  the  planning  standards  section  of  the  Emergency 
Management  Standard.  The  specific  areas  are  plans  for  continuity  of  government  (COG), 
continuity  of  operations  (COOP),  and  emergency  response/operations,  communications, 
and  recovery.289 

Comparing  the  results  of  the  DHS  OIG  and  EMAP  assessments  highlights  the 
organizations’  different  approaches.  The  OIG’s  focus  tends  to  follow  accounting 
principles  and  standards  while  looking  at  organizational  strategic  plans  and  general 
performance.  The  EMAP  assessment  is  more  focused  on  the  various  programmatic 
mission  requirements.  Both  perspectives  are  important  for  analyzing  performance,  and  a 
combined  approach  would  provide  a  more  comprehensive  assessment. 

There  are  also  challenges  to  identify  and  collect  proper  metrics.  Measurements 
must  be  tied  to  processes  or  outcomes  of  identified  goals  and  objectives  through 
organizational  strategic  plans.  However,  not  all  of  the  identified  metrics  will  be  owned  or 
generated  by  the  organization  desiring  the  data,  and  other  organizations  or  departments 
may  not  always  be  willing  to  provide  the  needed  data.  Another  trial  is  finding  a 
measurement  that  will  provide  the  appropriate  feedback  within  the  required  period  of 
time.  Taking  annual  measurements  will  not  allow  for  timely  policy  or  process  corrections 
and  would  hamper  management  effectiveness.  Measurements  that  fluctuate  on  a  daily  or 
weekly  timeframe,  on  the  other  hand,  are  perhaps  too  specific  to  provide  meaningful  and 
actionable  data. 

The  DOD  established  guidance  for  implementing  their  performance  management 
system  to  ensure  “a  common  set  of  standards  for  training,  certification,  deployment,  and 
operational  approaches  that  apply  a  common  body  of  knowledge  for  implementation  and 
execution  of  CPI/LSS  (continuous  process  improvement/lean  six  sigma  are  two 
disciplines  for  perfonnance  management  systems)  across  DOD  organizations.”290 

289  Emergency  Management  Accreditation  Program,  Assessment  Analysis  Report,  January >  - 
December  2013  (Lexington,  KY:  Emergency  Management  Accreditation  Program,  2014),  6, 
http://www.emaponline.org/index.php/root/assessor-resources/47-2013-assessment-analysis-report-l, 
Accessed  November  8,  2014. 

290  Department  of  Defense  Instruction  5010.43,  Implementation  and  Management  of  the  DOD-Wide 
Continuous  Process  Improvement/Lean  Six  Sigma  (CPI/LSS)  Program,  July  17,  2009,  6.c. 
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B. 


ALIGNMENT 


The  current  federal  performance  reporting  mechanism  for  all  SLTT  jurisdictions 
consists  of  three  main  products:  the  SPR  (required  annually  with  stakeholder  input  from 
jurisdictions  within  the  state),  the  THIRA  (also  required  annually  and  with  stakeholder 
input),  and  reports  required  by  individual  grant  programs  exceeding  $100,000  per  project 
or  grant  period — this  report  is  submitted  on  a  government-wide  form  called  the 
Perfonnance  Progress  Report  (SF-PPR). 

The  foundation  of  the  performance  management  system  resides  in  the  DHS’s  core 
strategic  documents:  the  Strategic  Plan,  Quadrennial  Homeland  Security  Review,  Annual 
Performance  Report,  and  National  Preparedness  Report.  This  foundation  is  represented 
in  Figure  4. 
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We  will  lead  efforts  to  achieve  a  safe,  secure,  and  resilient  homeland.  We  will  counter 
terrorism  and  enhance  our  security;  secure  and  manage  our  borders;  enforce  and 
administer  our  immigration  laws;  protect  cyber  networks  and  critical  infrastructure;  and 
ensure  resilience  from  disasters.  We  will  accomplish  these  missions  while  providing 
essential  support  to  national  and  economic  security  and  maturing  and  strengthening  both 
the  Department  of  Homeland  Security  and  the  homeland  security  enterprise. 


Transparency 


Public  Participation 


Collaboration 


Figure  4.  Foundation  of  DHS  Strategic  Planning291 


291  Data  derived  from  Department  of  Homeland  Security,  The  2014  Quadrennial  Homeland  Security’ 
Review ,  http://www.dhs.gov/sites/default/files/publications/qhsr/2014-QHSR.pdf.  and  Department  of 
Homeland  Security,  Fiscal  Years  2014-2018  Strategic  Plan, 

http://www.dhs.gov/sites/default/files/publications/FY14-18%20Strategic%20Plan.PDF. 
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It  is  essential  for  PPD-8  and  all  of  its  directed  guidance  documents  to  align  in 
order  to  properly  measure  and  evaluate  perfonnance.  An  enterprise’s  key  elements  (its 
people,  processes,  customers,  strategy,  and  leadership)  must  have  synergy  or  unity  of 
effort  to  realize  established  strategic  goals  and  objectives.  To  be  successful,  alignment 
occurs  in  two  key  dimensions:  vertically  and  horizontally.  The  chart  in  Figure  5  depicts 
the  essential  elements  of  the  vertical  and  horizontal  alignments. 
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Create  a  shared  strategic  vision  for 
the  organization 
Develop  a  deployment  plan  to 
translate  the  strategic  plan  into 
action 

Identify  critical  success  factors 


•  Identify  core  processes 

•  Redesign  key  processes 

•  Continuously  improve  processes 

•  Measure  and  monitor  results 


Identify  key  strategic  work  processes 
Align  all  organizational  work  with  strategic  vision 
Design  an  ongoing  process  for  systematic  review 
Measure  and  monitor  results 


■  Identify  corporate  and  individual 
competencies  necessary  to  achieve  the 
strategic  vision 

■  Identify  current  competency  levels 

•  Design  and  employ  plans  to  “close  gaps 


•  Identify  critical  “strategic"  customers 

•  Identify  customer  delight  factors 

•  Create  the  organizational  capability 
and  infrastructure  to  continuously 
gather  “actionable"  customer  data 
and  use  it  to  drive  process 
improvement  and  design 

•  Measure  and  monitor  results 


Design  and  align  reward  and 
recognition  systems  with 
organizational  goals  and  desired 
outcomes 

Measure  and  monitor  results 


Figure  5.  Vertical  and  Horizontal  Strategic  Alignment292 


292  George  Labovitz  and  Victor  Rosansky,  The  Power  of  Alignment — How  Great  Companies  Stay 
Centered  and  Accomplish  Extraordinary’  Things  (New  York:  John  Wiley  &  Sons,  1997),  232. 
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Vertical  alignment  ensures  all  members  of  the  enterprise  are  linked  to  the  strategy 
and  drives  an  opportunity  for  involvement  at  all  levels;  horizontal  alignment  links  the 
customer  to  the  business  processes  and  ensures  the  outcome  meets  the  recipients’ 
expectation.293  In  terms  of  national  preparedness,  strategic  policy  and  guidance  mandates 
that  the  customer  is  integrally  involved  in  not  only  providing  input  to  the  process,  but 
also  as  a  resource  provider.  PPD-8  directs  the  whole  community  as  an  integral  part  of  the 
process  “to  build  and  sustain  the  capabilities  necessary  to  prevent,  protect  against, 
mitigate  the  effects  of,  respond  to,  and  recover  from  those  threats  that  pose  the  greatest 
risk  to  the  security  of  the  Nation.”294 

One  way  to  align  the  goals  and  objectives  to  DHS  is  to  complete  a  matrix  similar 
to  the  one  shown  in  Figure  6.  Using  this  type  of  approach  will  illuminate  how  the  many 
SLTT  programs  fit  into  the  national  strategic  objectives.  Completing  this  type  of  activity 
should  also  include  the  many  local  and  state  stakeholders  (whole  community)  who  play  a 
role  in  the  various  homeland  security  and  emergency  management  aspects.  This  activity 
should  also  assist  the  SLTT  community  in  the  grant  application  and  justification  process. 


293  Ibid.,  26-37. 

294  Obama,  Presidential  Policy  Directive  8,  5. 
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STATE  HOMELAND  SECURITY  8.  EMERGENCY  MANAGEMENT  STRATEGY  ALIGNMENT  &  DEPLOYMENT  MATRIX 
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Figure  6.  Strategic  Alignment295 


295  Data  derived  from  Department  of  Homeland  Security,  Fiscal  Years  2012-2016  Strategic  Plan, 
http://www.dhs.gov/sites/default/fdes/publications/DHS%20Strategic%20Plan.pdf. 
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“Strategy  alignment  ensures  strategic  goals  and  priorities  cascade  across  the 
enterprise  with  supporting  implementation  and  initiative  priorities.”296  An  important 
aspect  of  strategy  alignment  is  performance  management  methods  that  make  visible 
performance  gaps  and  problem  resolution.  In  addition  transparently  identified  gaps,  a  key 
component  of  alignment  is  accountability  and  the  associated  dialogue.297 

DHS  provides  strategic  guidance  to  SLTT  jurisdictions  through  grant  funding 
opportunity  announcements  (FOA).  The  guidance  provides  funding  priorities  but  has  a 
generally  non-directive  approach,  using  language  such  as  “grantees  are  expected  to”  and 
“grantees  should.”298  It  is  unclear  what  actions  DHS  would  take  against  grantees  that  did 
not  follow  the  FOA’s  intent.  For  FY  2014,  the  priorities  were  determined  from  the  2013 
NPR,  and  included  “cyber  security,  recovery  of  core  capabilities,  integration  of 
individuals  with  functional  and  access  needs,  enhanced  resilience  of  infrastructure 
systems,  and  matured  public-private  partnerships.”299  Since  the  NPR  is  still  developing 
(it  is  only  on  its  third  annual  iteration),  using  the  report  for  detennining  funding  priorities 
is  new  for  FY  2014.  To  complement  the  selected  annual  grant  priorities,  25  percent  of  the 
total  HSGP  funding  allocation  is  required  to  be  directed  toward  law  enforcement 
terrorism  prevention  activities.  In  addition,  since  80  percent  of  the  HSGP  authorization  to 
each  state  must  be  passed  to  local  units  of  government,  there  is  a  built-in  whole 
community  aspect  to  this  grant  program. 

Analyzing  the  DHS  funding  priorities  from  FY  2008  through  FY  2014  leads  to 
two  distinct  observations.  First,  while  the  priorities  are  left  purposefully  vague  for  SLTT 
flexibility,  the  years  from  2008  through  2013  revealed  little  specificity  in  goals  and 
objectives.  In  addition,  there  seemed  to  be  little  continuity  through  the  years,  or  reasoning 
for  priorities.  Second,  the  2014  funding  priorities  based  on  the  NPR  are  trending  toward 


296  Department  of  Defense  Instruction  5010.43,  Implementation  and  Management  of  the  DOD-Wide 
Continuous  Process  Improvement/Lean  Six  Sigma  (CPI/LSS)  Program,  July  17,  2009,  6.b. 

297  Ibid. 

298  U.S.  Department  of  Homeland  Security,  Funding  Opportunity  Announcement  (FOA),  FY  2014, 
Homeland  Security  Grant  Program  (HSGP),  March  18,  2014.  http://www.fema. gov/media- 
library/asset  s/documents/92397. 

299  Ibid.,  33. 
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slightly  better  fidelity  and  specificity.  This  fidelity  will  continue  to  improve  in 
conjunction  with  reporting  system  development,  but  standardization  and  unreliable  state 
self-reporting  will  continue  to  be  an  issue  until  the  process  is  enhanced.  Funding 
priorities  for  FY  2008  through  2014  are  outlined  in  Figure  7. 
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Figure  7.  Homeland  Security  Grant  Program  Funding  Priorities  FY  08-14300 


PPD-8  directed  an  “integrated,  all-of-nation,  capabilities-based  approach  to 
national  preparedness.”301  The  latest  HSGP  funding  guidance  calls  for  programs  that  are 
based  on  risk-driven,  capabilities-based  strategic  plans  that  address  both  the  sustainment 
of  current  capabilities  and  the  mitigation  of  capability  gaps.  The  guidance  further  states 
that  “grantees  are  to  prioritize  their  funding  in  order  to  address  gaps  in  achieving 


300  Data  obtained  from  respective  Funding  Opportunity  Announcements  &  Guidance  from  the  FEMA 
Grants  website:  https://www.fema.gov/preparedness-non-disaster-grants 

301  Obama,  Presidential  Policy  Directive  8,  1. 
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capability  targets  set  through  the  annual  THIRA,  and  identified  through  the  annual 
SpR.”302 

All  states  and  territories  are  provided  funding  authorization  according  to  three 
criteria:  legislative  minimum  amounts,  anticipated  program  effectiveness,  and  DHS’s  risk 
methodology.  In  addition,  they  are  expected  to  consider  funding  priorities  identified  in 
the  previous  year’s  NPR.303 

This  guidance  shapes  the  questions  that  determine  funding  allocation.  For  the 
SHSP,  the  baseline  funding  formula  for  2014  was  0.35  percent  of  the  total  allocation  to 
52  states  and  territories,  and  0.08  percent  to  the  remaining  four  territories  comprising 
approximately  19  percent  of  the  total  budget.304  The  DHS  determines  relative  risk  to  a 
jurisdiction  by  analyzing  the  area’s  population,  critical  infrastructure,  and  economic 
security.  Finally,  the  effectiveness  component  is  based  on  how  project  description  in  the 
investment  jurisdiction  section  of  the  grant  application  aligns  with  the  gaps  identified  in 
the  state  THIRA  and  SPR.  Of  the  56  eligible  SHSP  grantees,  25  receive  funding  solely  on 
the  baseline  funding  formula.305  It  is  not  clear  or  readily  transparent  how  the 
effectiveness  and  the  DHS  risk  assessment  criteria  affect  or  influence  funding 
detenninations. 

Given  the  lack  of  available  preparedness  performance  criteria,  it  is  unlikely  that 
this  component  holds  significant  weight  within  the  analysis.  If,  however,  reliable  and 
meaningful  effectiveness  data  were  available,  this  component  could  become  a  key 
indicator  of  the  funding  allocation  equation.  Performance  data  could  help  answer  the 
question:  How  successful  were  the  grant  applicants  in  meeting  their  stated  goals  and 
objectives  during  prior  funding  years? 


302  U.S.  Department  of  Homeland  Security,  Funding  Opportunity  Announcement  (FOA),  FY  2014, 
Homeland  Security’  Grant  Program  (HSGP),  4,  http://www.fema. gov/media- 

library/asset  s/documents/92397. 

303  Ibid.,  57. 

304  FEMA,  FEMA  Grants  Program  Directorate  Information  Bulletin  #398a  (Washington,  DC: 
Department  ofHomeland  Security,  2014)  5,  http://www.fema.gov/media-library/assets/documents/97599. 

305  Ibid. 
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An  organization’s  ability  to  establish  performance  criteria  and  show  significant 
progress  toward  identified  targets  is  an  indicator  of  effectiveness.  A  related  question  to 
detennine  program  effectiveness  is:  How  did  the  grant  programs  meet  state  or  national 
priorities  as  identified  in  their  respective  strategic  plans?  Due  to  the  reduction  in  grant 
funding  over  the  past  six  years,  there  are  limited  funds  available  to  state  homeland 
security  programs  and  not  all  priorities  can  be  confronted.  It  is  therefore  essential  that 
those  limited  resources  are  used  to  address  the  strategic  priorities.  Finally,  the  question  of 
how  jurisdictions  are  able  to  sustain  and  validate  capabilities  through  training,  exercises, 
and  equipment  maintenance  and  replacement  can  be  addressed. 

Goals,  objectives,  and  priorities  can  be  aligned  through  a  detailed  planning 
process.  A  simple  table  identifying  DHS’s  strategy  and  the  SLTT  jurisdiction’s  correlated 
goals,  objectives,  and  performance  metrics  is  useful  to  show  aligned  missions  and 
programs.  A  conceptual  idea  of  this  output  is  displayed  in  Figure  8.  This  process  will 
facilitate  management  decisions  by  ensuring  that  management  understands  process 
efficiencies  and  effectiveness,  informs  jurisdictional  appropriators,  and,  with  additional 
process  development  and  analysis,  provides  data  to  infonn  Congress  of  preparedness 
grant  programs’  effectiveness. 
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Figure  8.  Alignment  of  Goals,  Objectives,  and  Metrics 


C.  PROCESS  ANALYSIS 

Evaluating  performance  is  more  than  just  outputs,  outcomes,  or  a  list  of 
capabilities.  In  reviewing  an  organization’s  ability  to  perform  its  mission,  there  are  many 
other  factors  that  directly  impact  effectiveness,  such  as  internal  decision-making 
processes,  leadership  qualities,  and  external  social  networks.306  For  this  reason,  looking 
at  the  entire  process  for  efficiencies  and  quality  production  will  affect  the  organization’s 
capacity  to  conduct  operations  and  improve  the  overall  results.  This  is  known  as  process 
analysis. 

Process  analysis  involves  a  detailed  review  of  the  horizontal  alignment  and  is 
commonly  known  as  the  “SIPOC,”  or  suppliers,  inputs,  process,  output,  and  customer. 


306  Alan  Kirschenbaum,  “Measuring  the  Effectiveness  of  Disaster  Management  Organizations,” 
International  Journal  of  Mass  Emergencies  and  Disasters  22  (March  2004):  81, 
http://www.ijmed.org/articles/389/download/. 
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These  elements  describe  the  flow  of  goods  and  services  through  an  organization  that 
eventually  lead  to  outcomes  and  recognized  success  levels.  For  this  analysis,  the  SIPOC 
model  is  reversed;  it  begins  with  perhaps  the  most  critical  element  in  the  chain:  the 
customer.  The  reverse  flow  follows  the  COPIS  model  as  illustrated  in  Figure  9. 


Customers  Outputs 


Process 
Requirements 


Inputs 


Suppliers 


Figure  9.  High  Level  Process  Map 


Like  the  vertical  and  horizontal  alignment  model,  the  COPIS  model  offers 
opportunities  to  identify  and  collect  key  metrics  and  perfonnance  measures  throughout 
the  entire  chain  of  events.  The  primary  identification  sources  for  these  performance 
indicators  are  typically  found  within  the  core  processes.  One  of  perfonnance 
measurement’s  main  challenges  is  collecting  data  that  supports  the  identified  metrics.  In 
some  cases,  the  effort  to  collect  data  or  measure  activities  will  exceed  the  department’s 
resources.  If  the  measure  is  critical  to  assessing  performance,  there  may  be  opportunities 
to  identify  other  measures  that  are  easier  to  collect  and,  together,  provide  performance 
measurement  indication.  Additionally,  the  organization  may  not  always  own  the  critical 
data,  and  may  need  to  enter  into  agreements  with  external  entities  to  access  the 
information.  In  other  cases,  data  requirements  within  the  department  will  be  available  but 
not  currently  collected. 

While  many  of  the  key  processes  will  align  with  the  mission  areas  identified  in 
DHS’s  Strategic  Plan,  there  will  also  be  unique  processes  and  mission  areas  within  each 
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unique  jurisdiction,  statutory  requirements,  and  risks.  Likewise,  there  will  be  DHS 
mission  areas  that  will  not  directly  apply  to  all  SLTTs. 

1.  Customer 

For  the  purposes  of  this  research  and  the  challenge  of  assessing  and  reporting  on 
national  preparedness,  the  customer  is  focused  primarily  on  congressional  appropriators 
and  congressional  committees  that  oversee  DHS.  For  the  purpose  of  a  process  analysis, 
the  customer  can  include  anyone  within  the  definition  of  whole  community,  as  all  would 
benefit  (receive  goods  and  services)  from  the  enterprise.  The  Office  of  Management  and 
Budget  (OMB)  defines  a  customer  indicator  as  a  type  of  measure  that  informs  the 
improvement  of  government’s  interaction  with  those  it  serves  or  regulates.307 

The  customer  is  generally  considered  the  area  of  primary  significance  in 
collecting  satisfaction  measures.  The  main  challenge  in  measuring  customer  satisfaction 
is  often  that  the  result  is  based  on  perception.  Since  the  customer  is  at  the  end  of  the 
goods-or-services  delivery  process,  rates  of  satisfaction  can  be  considered  a  reflection  of 
the  entire  process.  In  essence,  the  gap  between  what  the  organization  claims  are  its 
customer-oriented  goals  and  the  customer’s  perception  its  true  goal  achievement  fonn  the 
basis  for  a  measure  of  organizational  effectiveness.308 

2.  Inputs 

Inputs  are  the  resources  required  to  produce  a  good  or  service.  OMB  defines  an 
input  indicator  as  a  measure  that  indicates  the  consumption  of  expended  resources, 
especially  time  and/or  money.309  While  preparedness  is  “often  associated  with 
capabilities  and  training,  key  inputs  can  also  include  people,  equipment,  supplies,  and 
maintenance.”310  According  to  the  Government  Auditing  Standards,  “inputs  are  the 


307  OMB,  Circular  No.  A-ll,  Preparation,  Submission,  and  Execution  of  the  Budget,  Washington, 
DC:  The  White  House,  2014),  Part  6,  Section  200.21, 

http://www.whitehouse.gov/sites/default/files/omb/assets/al  l_current_year/al  l_2014.pdf. 

308  Kirschenbaum,  “Measuring  the  Effectiveness  of  Disaster  Management  Organizations,”  87. 

309  OMB,  Circular  No.  A-ll,  Part  6,  Section  200.21. 

310  Harrison,  “Rethinking  Readiness,”  42. 
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amount  of  resources  (in  terms  of  money,  material,  personnel,  etc.)  that  are  put  into  a 
program.  These  resources  may  come  from  inside  or  outside  the  entity  operating  the 
program.  Input  measures  can  have  a  number  of  dimensions,  such  as  cost,  timing,  and 
quality.  Examples  of  input  measures  are:  dollars  spent,  employee-hours  expended,  and 
square  feet  of  building  space.”311 

For  the  DOD,  the  current  balance  of  inputs  was  developed  over  decades  of 
experience;  these  inputs  have  produced  an  outcome  that  is  deemed  positive,  as  confirmed 
in  recent  military  operations.  The  challenge  for  the  DOD,  and  DHS,  is  to  produce  the 
desired  outcomes  while  ensuring  inputs  are  optimally  structured  and  efficient,  and  do  not 
waste  precious  resources.  Additionally,  in  a  dynamic  environment  such  as  war  or 
disaster,  what  has  traditionally  worked  in  the  past  may  not  always  work  for  future 
events.312 


3.  Process 

Process  is  generally  concerned  with  the  internal  transactions  required  to  produce 
an  output.  There  are  many  measures  within  the  process  that  can  provide  indications  of 
efficiency.  OMB  defines  an  efficiency  indicator  as  a  type  of  measure,  specifically  “a  ratio 
of  a  program  activity  inputs  (such  as  costs  or  hours  worked  by  employees)  to  its  outputs 
or  outcomes.”313  Efficiency  indicators  reflect  the  resources  used  to  achieve  outcomes  or 
produce  outputs.  Process  measures  can  be  as  simple  as  the  cost  to  produce  a  single  unit  of 
output.  In  other  circumstances,  it  is  more  useful  to  find  effective  practices  and  then  look 
for  lower-cost  delivery  options.314  These  metrics  provide  program  managers  and  process 
owners  insight  on  where  bottlenecks  or  delays  in  process  flow  occur.  The  diagram  in 
Figure  10  shows  an  example  of  a  simple  notional  process  that  would  track  the  cost  of 
human  resources  and  show  areas  for  potential  improvement  in  wasted  or  idle  time.  These 
metrics  are  types  of  process  indicators,  which  they  OMB  define  as  types  of  measure  that 


311  GAO,  Government  Auditing  Standards,  131. 

312  Harrison,  “Rethinking  Readiness,”  62. 

313  OMB,  Circular  No.  A-ll,  Part  6,  Section  200.21. 

314  Ibid. 
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indicate  how  well  a  procedure,  process  or  operation  is  working,  (e.g.,  timeliness, 
accuracy,  or  completeness).315 


315  Ibid. 
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Figure  10.  Simple  Process  Flow  Diagram 
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Process  flow  diagrams  can  represent  simple  or  complex  business  interactions. 
Using  Michael  Hammer’s  three  main  process  categories  of  governing,  core,  and  enabling, 
we  can  begin  to  identify  the  key  elements  contributing  to  DHS’s  mission.  The  chart  in 
Figure  1 1  shows  a  graphic  representation  of  the  key  elements  in  each  process  category. 
Each  category  and  its  elements  function  together  to  realize  the  desired  goal,  objective,  or 
outcome.  It  is  therefore  essential  to  identify  the  elements  within  each  of  the  categories 
and  analyze  their  respective  processes  and  interrelationships. 
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Figure  11.  Three  Primary  Process  Categories316 


4.  Outputs 

Outputs  are  generally  the  products  or  services  the  customer  is  demanding.  In  the 
manufacturing  sector,  outputs  can  be  measured  by  simply  counting  the  number  of 
widgets  produced.  There  can  also  be  associated  metrics,  such  as  the  time  required  to 
produce  a  widget,  the  number  of  defects  discovered  at  the  end  of  the  production  line,  or 
the  production  resources’  efficiency.  Outputs  represent  “the  quantity  of  goods  or  services 
produced  by  a  program.  For  example,  an  output  measure  for  a  job  training  program  could 


316  Michael  Hammer  and  Lisa  Hershman,  Faster  Cheaper  Better:  The  9  Levers  for  Transforming  How 
Work  Gets  Done  (New  York:  Crown,  2010),  19. 
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be  the  number  of  persons  completing  training,  and  an  output  measure  for  a  safety 
program  could  be  the  number  of  safety  inspections  completed.”317 

OMB  describes  an  output  indicator  as  a  type  of  measure,  specifically  “the 
tabulation,  calculation,  or  recording  of  activity  or  effort,  usually  expressed 
quantitatively.”318  They  also  distinguish  that  an  output  indicator  should  also  have  a 
“reasonable”  connection  to  outcomes.  Therefore,  output  indicators  should  be  based  on 
evidence  supporting  the  relationship  between  outputs  and  outcomes.319 

In  the  mission  areas  of  readiness  or  preparedness  good  output  measures  can  be 
challenging  to  develop  and  collect.  Consequently,  inputs  are  often  used  in  lieu  of  outputs 
because  money  spent  or  the  numbers  of  widgets  purchased  are  easy  to  document  and 
track.  In  his  report  on  assessing  military  readiness,  Todd  Harrison  describes  the 
challenges  associated  with  using  inputs  over  outputs  as  the  primary  measurement  tool. 

In  the  DOD,  readiness  inputs  are  often  used  as  proxy  measures  for  the 
output — forces’  ability  to  perform  their  assigned  missions.  But  an  implicit 
assumption  in  this  approach  is  that  changes  in  the  inputs  will  result  in 
corresponding  changes  in  the  outputs.  Moreover,  it  assumes  that  the  input 
target  levels  set  by  the  military  are  optimal  to  achieve  the  types  and  levels 
of  readiness  required  by  defense  strategy. 320 

These  assumptions  show  the  limits  input  measures  alone  can  have  on  managers 
and  leaders  in  using  this  type  of  data  to  make  informed  decisions  and  reliably  track 
organizational  perfonnance. 

5.  Outcomes 

Outcomes  are  a  program’s  accomplishments  or  results.  Outcomes  may  be 
influenced  by  a  variety  of  external  factors  to  include  cultural  differences  or  a  dramatic 
change  in  national  or  local  economic  conditions.  “Outcome  measures  show  the  progress 


317  U.S.  Government  Accountability  Office  (GAO),  Government  Auditing  Standards  2011  Revision, 
(GAO-12-33 1G),  (Washington,  DC:  U.S.  Government  Accountability  Office,  2011),  132, 

http ://  www.  gao .  go  v/pro  duct  s/ GAO  - 1 2-  3  3 1 G . 

318  OMB,  Circular  No.  A-ll,  Part  6,  Section  200.21. 

319  Ibid. 

320  Harrison,  “Rethinking  Readiness,”  41-42. 
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made  in  achieving  the  stated  program  purpose.  Outcomes  also  include  unexpected  and/or 
unintentional  effects  of  a  program,  both  positive  and  negative. 32  lu 

An  intermediate  outcome  indicator  is  a  type  of  measure  that  indicates  progress 
against  an  intermediate  outcome.  The  intermediate  outcome  then  contributes  to  an 
ultimate  outcome,  such  as  the  percentage  of  schools  adopting  effective  literacy  programs, 
compliance  levels,  or  the  rate  of  adoption  of  safety  practices.322  Outcome  indicators 
measure  progress  against  the  intended  result  of  a  program  and  changes  in  conditions  the 
government  is  trying  to  influence.323  Examples  of  outcome  measures  can  be:  the 
retention  rate  of  fully  trained  staff;  the  rate  of  change  in  safety  violations  or  incidents;  or 
the  success  rate  of  corrective  actions  on  previously  identified  problems. 

D.  ASSESSMENT  AND  REPORTING 

The  benefits  and  challenges  of  self-assessment  in  the  homeland  security  enterprise 
were  discussed  previously  in  this  Chapter  in  section  A.  “Systematically  using  self- 
reported  data  or  expert  judgment  in  assessments  is  difficult,  but  processes  can  be 
designed  to  minimize  subjectivity  or  other  skewing  effects.”324  One  common  method  to 
inculcate  standardization  and  reliability  into  a  self-assessment  system  is  to  add  oversight. 
There  are  three  examples  of  oversight  that  utilize  an  external  fonn  of  assessment:  the 
EMAP,  the  Baldrige  Performance  Excellence  Program,  and  the  USAF. 

EMAP  uses  a  self-assessment  process  to  evaluate  the  efficiency  and  effectiveness 
of  emergency  management  programs.  To  mitigate  the  challenges  of  self-assessing  and 
reporting  the  EMAP  provides  an  independent  and  on  site  review  of  all  documentation  and 
programs.  The  goal  of  accreditation  “is  to  evaluate  an  emergency  management  program’s 
organization,  resources,  plans,  and  capabilities  against  current  standards  to  increase 
effectiveness  in  protecting  residents’  lives  and  property.”325 

321  GAO,  Government  Auditing  Standards  2011  Revision,  132. 

322  OMB,  Circular  No.  A-l  ,  Part  6,  Section  200.21. 

323  Ibid. 

324  Jackson,  The  Problem  of  Measuring  Emergency  Preparedness,  20-21. 

325  Emergency  Management  Accreditation  Program  (EMAP),  Candidate’s  Guide  to  Accreditation, 
(Lexington,  KY:  Emergency  Management  Accreditation  Program,  2011),  1. 
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EMAP  establishes  the  minimum  acceptable  performance  criteria  for  an 
emergency  management  program  and  also  establishes  a  means  for  strategic 
improvement.326  The  self-assessment  component  is  designed  to  identify  and  correct  any 
deficiencies  against  the  published  standards.  Then  an  external  team  is  assigned  to  conduct 
an  on-site  assessment  of  ah  the  standards  and  validate  the  program’s  self-reporting.  There 
are  ten  steps  to  attain  EMAP  accreditation: 

1 .  Request  infonnation 

2.  Program  assessment  tool  subscription 

3.  Self-assessment  and  documentation 

4.  Application  for  accreditation 

5.  Preparation  for  on-site  assessment 

6.  On-site  assessment  and  report 

7.  Committee  review  and  recommendation 

8.  Commission  consideration  of  committee  recommendations 

9.  Accreditation  certificate  presentation 

10.  Accreditation  process  critique327 

Once  accredited,  the  program  is  required  to  submit  an  annual  update  on  efforts 
taken  to  remain  EMAP  compliant  until  the  next  external  assessment,  which  is  due  after 
five  years.  This  is  similar  to  the  USAF’s  compliance  assessment  approach. 

The  USAF  uses  an  on-site  inspection  as  the  assessment  system’s  external 
component.  The  USAF  inspection  system’s  purpose  is  threefold:  “to  enable  and 
strengthen  effectiveness  and  efficiency;  to  promote  improved  unit  perfonnance  and 
management  excellence  up  and  down  the  chain  of  command;  and  to  identify  issues 
interfering  with  effectiveness,  efficiency,  compliance,  readiness,  perfonnance,  and 


326  Emergency  Management  Accreditation  Program,  A  Framework  for  Assessing  Regional 
Preparedness,  (Lexington,  KY:  Emergency  Management  Accreditation  Program,  2006),  1, 
http  ://www.  emaponline .  org/?2 5 6 . 

327  EMAP,  Candidate’s  Guide  to  Accreditation,  3. 
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management  excellence.”328  The  assessment  system  provides  senior  leaders  an 
independent,  standardized  review  of  all  Air  Force  units  and  validates  the  unit’s  self- 
assessment  processes.329 

One  of  the  key  benefits  of  the  independent  is  that  it  brings  in  assessors  who 
possess  insight  and  experience  from  visiting  many  units  that  manage  the  same  functions. 
Not  only  do  they  have  the  most  up-to-date  knowledge  on  guidance  and  policy,  they  are 
also  able  to  analyze  and  evaluate  how  disparate  organizations  approach  and  resolve 
common  challenges.  The  assessors  serve  as  subject-matter  experts,  but  also  as  collectors 
and  distributors  of  established  good  practices  and  lessons  learned.330  The  objectivity  and 
fresh  perspective  that  outsiders  bring  to  an  organization  are  considered  important  benefits 
of  any  assessment  system.331 

The  USAF  also  conducts  a  “management  inspection”  as  part  of  their  overall 
assessment  portfolio.  This  inspection  consists  of  seven  major  categories:  strategic 
planning,  organizational  management,  customers,  process  operations,  resources,  data- 
driven  decisions,  and  organizational  performance.332  These  seven  categories  are  similar 
in  focus  to  the  internationally  accepted  “Baldrige  Criteria  for  Perfonnance  Excellence 
categories:  leadership;  strategic  planning;  customer  focus;  measurement,  analysis,  and 
knowledge  management;  workforce  focus;  operations  focus;  and  results.”333  The  table  in 
Figure  12  compares  the  major  categories  within  each  assessment  for  the  USAF  and 
Baldrige  assessment  programs. 


328  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  7-8. 

329  Ibid.,  19. 

330  Camm  et  at,  Charting  the  Course,  46. 

331  Ibid.,  45. 

332  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  38-39. 

333  National  Institute  of  Standards  and  Technology,  U.S.  Department  of  Commerce,  Baldrige 
Performance  Excellence  Program,  January  2013,  ii. 
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Figure  12.  Comparison  of  Air  Force  and  Baldrige  Criteria334 


The  Baldrige  Performance  Excellence  Program  uses  tools,  assessment  criteria, 
and  standards  to  facilitate  a  robust  self-assessment  program.  In  addition  to  the  national 
program,  there  are  also  regional,  state,  and  local  programs  that  can  provide  assistance  to 
organizations.  In  order  to  receive  an  award  for  performance  excellence  at  the  local,  state, 
or  national  levels,  however,  an  on-site  assessment  by  trained  and  credentialed  assessors  is 
required.  The  state  benefits  of  applying  for  the  Malcolm  Baldrige  National  Quality 
Award  are: 

•  Accelerated  improvement  efforts:  The  application  process  itself 
accelerates  and  goes  beyond  internal  self-assessments  by  bringing  a 
rigorous,  objective,  external  viewpoint  to  an  organization’s  internal 
improvement  process. 

•  Energized  employees:  Pursuing  a  common  goal  motivates  employees, 
resulting  in  energized  improvement  efforts. 

•  An  outside  perspective:  Applying  for  the  Baldrige  Award  is  an 
opportunity  to  have  a  team  of  experts  examine  your  organization 
objectively  and  identify  strengths  and  opportunities  to  improve.  Five  to 
eight  trained  experts  from  an  independent  Board  of  Examiners  spend  a 
minimum  of  300  hours  reviewing  each  application,  and  site-visited 
applicants  receive  more  than  1,000  hours  of  review. 


334  Air  Force  Criteria  are  obtained  from  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  Air 
Force  Publication  90-201  (Washington,  DC:  Secretary  of  Air  Force  (IG),  2013).  The  Baldrige  Criteria  are 
obtained  from  “Criteria  for  Performance  Excellence,  2013-2014”  Baldrige  Performance  Excellence 
Program,  http://www.nist.gov/baldrige/publications/archive/2013_2014_business_nonprofit_criteria.cfm. 
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•  Learning  from  the  feedback:  Each  applicant  receives  an  objective,  non- 
prescriptive,  written  feedback  report  of  its  strengths  and  opportunities  for 
improvement  based  on  an  examination  of  its  award  application  against  the 
Baldrige  Criteria  for  Performance  Excellence.  Organizations  often  use  the 
report  in  their  strategic  planning  to  focus  on  their  customers  and  improve 
results,  as  well  as  to  help  energize  and  guide  improvement  efforts. 

•  Aligned  efforts  and  resources:  The  linkages  among  the  requirements  of  the 
seven  Baldrige  Criteria  categories  help  your  organization  achieve  better 
coordination  and  consistency  among  plans,  processes,  information, 
resource  decisions,  actions,  results,  analysis,  and  learning.335 

E.  TRAINING 

In  order  to  effectively  work  within  a  performance  management  system,  all  levels 
of  the  enterprise  must  have  trained  and  knowledgeable  staff  to  advise  and  assist  in 
strategic  and  performance  planning  activities.  The  functions  equivalent  to  a  perfonnance 
improvement  officer  (PIO)  within  the  federal  system  requires  highly  specific  skills  and 
expertise.  There  are  many  private  institutions  that  provide  standardized  and  credentialed 
training  and  education  to  meet  PIO  requirements.  In  addition  to  the  PIO,  the  SLTT 
community  will  also  need  to  expand  their  knowledge,  skills,  and  abilities  to  assess 
program  effectiveness. 

Currently  DHS,  through  the  Emergency  Management  Institute,  offers  a  three-day 
training  course,  designated  the  E0122  Emergency  Management  Accreditation  Program. 
The  training  is  designed  to  inform  emergency  management,  preparedness,  and  homeland 
security  personnel  with  information  regarding  the  accreditation  process  and  assessment 
system  using  the  established  standards.336  After  passing  the  course  exam,  participants  are 
eligible  to  serve  as  a  jurisdiction’s  or  program’s  accreditation  manager,  and  can  also 
become  voluntary  assessors  for  EMAP  if  they  meet  additional  experience  and  knowledge 
requirements.337  The  National  Domestic  Preparedness  Consortium  partners  are  also  well 


335  For  information  about  applying  for  the  Malcom  Baldrige  National  Quality  Award,  please  see 
http://www.nist.gov/baldrige/enter/apply.cfm. 

336  Emergency  Management  Institute  (EMI),  Fiscal  Year  15  Training  Catalog,  Train  Exercise, 
Educate,  48 — 49,  training.fema.gov/EMICourses/EMICatalog.asp. 


337  Ibid. 
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suited  to  expand  their  extensive  training  and  education  portfolio  to  include  perfonnance 
management  or  assessment  education. 

F.  ORGANIZATIONAL  ROLES  AND  RESPONSIBILITIES 

The  9/11  Commission  Act  of  2007  requires  the  DHS  OIG,  to  audit  individual 
states’  and  territories’  management  of  State  Homeland  Security  Program  and  Urban 
Areas  Security  Initiative  grants.  These  audits’  objectives  are  to  detennine  if  grant 
recipients  have  distributed  and  spent  grant  funds  effectively  and  efficiently,  and  if  they 
have  done  so  in  compliance  with  laws  and  regulations,  state  homeland  security  strategies, 
and  other  applicable  plans.  In  addition,  the  OIG  assesses  the  extent  to  which  funds 
awarded  help  meet  the  National  Preparedness  Goal  objectives.  For  all  deficient  areas 
identified,  the  OIG  provides  recommendations  designed  to  strengthen  program 
management,  performance,  and  oversight  of  the  jurisdiction’s  program.338 

The  latest  NPR  highlighted  a  potential  issue  with  divergent  perceptions  regarding 
roles  and  responsibilities  in  the  homeland  security  enterprise.  One  of  the  questions 
involved  a  local  perspective  on  federal  versus  state  roles  in  addressing  gaps  in  the  3 1  core 
capabilities.  State  responses  across  all  the  core  capabilities  ranged  from  53  percent  to  91 
percent  in  the  categories  of  “entirely  a  state  responsibility”  or  “mostly  a  state 
responsibility.”339  DHS’s  comments  regarding  the  results  were:  “These  insights  reflect 
state  and  territory  observations  from  the  State  Preparedness  Report  and  may  differ  from 
Federal  perspectives  on  who  bears  more  responsibility  for  addressing  identified  gaps. 
However,  these  observations  promote  dialogue  among  Federal,  state,  local,  tribal,  and 
territorial  partners  regarding  expected  roles  and  responsibilities.”340  This  finding  is 
critical  to  the  entire  preparedness  goal.  Without  clearly  understanding  who  owns  the 
responsibility  for  closing  capability  gaps,  how  can  the  homeland  security  enterprise 
efficiently  close  those  gaps  without  substantial  duplication  of  effort? 


338  DHS  Office  of  Inspector  General,  Countering  Terrorism,  14. 

339  FEMA,  National  Preparedness  Report,  11. 

340  Ibid. 
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The  EMAP  identifies  the  selection  of  an  accreditation  manager  as  a  critical  senior 
management  decision.  The  accreditation  manager  ensures  the  data  collection  and  results 
of  all  the  organization’s  activities,  plans,  and  procedures  meet  the  established  standards. 
Compliance  detennination  is  made  through  detailed  analysis  and  documentation.  This 
position  is  best  served  by  someone  who  can  provide  long-tenn  continuity  to  the  position 
for  consistent  and  reliable  results.341 

Metrics,  Data,  and  Measurement 

The  collection  of  measurements  is  a  detailed  and  critical  process.  As  with  the 
threat  intelligence  cycle,  the  sole  act  of  collecting  data  does  not  provide  sufficient  context 
for  managers  to  make  infonned  decisions.  Data  must  be  analyzed,  compared,  sorted  or 
otherwise  manipulated  to  create  information.  This  new  infonnation  is  then  synthesized 
with  other  information,  or  experience,  to  create  knowledge.  Over  time,  knowledge  is 
assimilated  into  managerial  and  organizational  wisdom,  which  is  manifested  in  the 
leaders’  decisions.342 

Another  way  to  impart  context  to  data  is  to  use  a  balanced  approach.  The 
recognized  standard  for  balance  in  a  measurement  system  is  the  Balanced  Scorecard, 
which  uses  different  viewpoints  to  analyze  an  organization’s  outcome  or  output.  The  four 
main  categories  used  by  Kaplan  and  Norton  to  develop  a  balanced  perspective  on 
performance  are  customer,  financial,  internal  business  processes,  and  learning  and 
growth.343  Applying  different  sets  of  data,  from  different  perspectives  but  related  to  a 
specific  process,  helps  provide  additional  context  and  facilitates  analysis.  This  kind  of 
data  can  also  be  described  as  contextual  indicators,  or  data  that  provides  situational 
information  for  the  purpose  of  better  understanding  how  trends  are  related  to  a  goal  or  a 
program.344 


341  Emergency  Management  Accreditation  Program,  Candidate’s  Guide  to  Accreditation,  2. 

342  Dean  R.  Spitzer,  Transforming  Performance  Measurement,  Rethinking  the  Way  We  Measure  and 
Drive  Organizational  Success  (New  York:  American  Management  Association,  2007),  105-107. 

343  Robert  S.  Kaplan  and  David  P.  Norton,  Translating  Strategy  into  Action:  The  Balanced  Scorecard 
(Boston:  Harvard  Business  School  Press,  1996),  43. 

344  OMB,  Circular  No.  A-ll,  Part  6,  Section  200.21. 
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The  primary  challenge  to  understanding  national  preparedness  perfonnance  is  the 
data  availability  and  collection  gap  that  is  inherent  within  the  “whole  community.” 
Current  laws  mandate  federal  agencies  comply  with  the  collection  and  reporting 
requirements  for  performance  management.  These  directives,  however,  do  not  apply  to 
SLTT  jurisdictions.  DHS,  however,  through  grant  guidance,  does  require  SLTT  entities 
to  provide  investment  justifications  for  expenditures  during  the  application  process, 
progress  reports  throughout  the  fiscal  year,  and  the  annual  SPR  and  the  THIRA  reports. 

If  the  whole  community  is  deemed  an  essential  element  of  the  homeland  security 
enterprise  and  is  critical  to  successfully  meeting  national  goals  and  objectives,  then  there 
needs  to  be  a  focused  and  substantial  effort  applied  toward  identifying  and  collecting 
meaningful  data  that  can  help  drive  policy,  initiatives,  and  resource  allocation.  “Whole 
community  partners  offer  programs  and  initiatives  that  contribute  to  the  core  capabilities 
outlined  in  the  Goal.  However,  in  many  cases,  measures  and  metrics  do  not  yet  exist  to 
gauge  perfonnance,  either  quantitatively  or  qualitatively,  over  time.  PPD-8  envisions  a 
National  Preparedness  System  that  includes  a  comprehensive  approach  to  assessments 
using  clear,  objective,  and  quantifiable  performance  measures.”345 

Collecting  good  data  is  an  arduous  and  intensive  task.  As  a  result,  many 
organizations  will  select  measures  that  are  easy  to  obtain  but  may  not  provide  the  data 
needed  to  make  infonned  decisions.  Collecting  meaningful  measures  is  often  a  more 
complex  endeavor  and  will  demand  the  commitment  of  time  and  resources.346 

The  GPRAMA  describes  several  considerations  in  determining  accurate  and 
reliable  data  sources.347  First,  the  intended  use  for  the  data  and  how  accurate  the  data 
needs  to  be  to  measure  the  progress  must  be  detennined.  Sometimes,  using  already 
available  data  is  preferred,  but  it  does  not  make  sense  to  measure  with  a  micrometer  when 
a  yardstick  will  provide  the  needed  information.  This  consideration  will  ensure  data 
collection  resources  are  used  frugally  and  economically. 

345  FEMA,  National  Preparedness  Report. 

346  Jackson,  The  Problem  of  Measuring  Emergency  Preparedness,  20-21. 

347  Government  Performance  and  Results  Act  (GPRA)  Modernization  Act  of  2010,  Pub.  L.  No.  111- 
352,  124  Stat.  3866  (2011) 


107 


The  next  step  is  to  identify  the  data  source  or  sources.  The  goal  for  identifying 
data  sources  is  to  minimize  the  collection  process’s  effort.  Adding  steps  to  existing 
processes  in  order  to  collect  data  is  not  ideal,  but  it  is  often  necessary,  particularly  if  data 
collection  was  not  previously  part  of  the  agency  culture.  Once  the  data  is  collected,  there 
must  be  a  means  to  verily  and  validate  the  data  values. 

The  GAO  defines  verification  as  “a  process  of  checking  or  testing  performance 
information  to  assess  other  types  of  errors,  such  as  errors  in  keying  data.”348  Validation  is 
defined  as  “an  effort  to  ensure  that  data  are  free  of  systematic  error  or  bias,  and  that  what 
is  intended  to  be  measured  is  actually  measured.”349  This  step  is  critical  to  avoid  the  old 
axiom  “garbage  in,  garbage  out.”  Since  the  data  is  instrumental  to  determine  funding 
priorities  and  other  strategic  decisions,  good  data  feeding  into  the  decision  process  is 
vital.  Verification  and  validation  support  the  data’s  accuracy  and  reliability  and  instill 
confidence  in  management  for  credible  and  transparent  decisions. 

Finally,  any  limitations  to  the  available  data  will  need  to  be  understood  and  a 
strategy  must  be  developed  to  either  compensate  for  the  limitation  or  provide  a  caveat  at 
the  data  presentation.  There  are  limitations  to  all  collected  data  that  can  lead  to  poor  or 
uninformed  decisions,  which  result  in  lower  performance  or  inaccurate  perfonnance 
assessments.  Significant  data  limitations  should  be  documented  on  the  assessment  with 
an  assessment  of  potential  impacts  for  the  organization  as  well  as  mitigation  strategies  to 
counter  the  limitations.  Data  limitations  can  include  imprecise  measurements,  incomplete 
data,  an  inconsistent  data  collection  process,  or  data  collection  rates  that  do  not  allow 
timely  strategy  adjustments  or  decisions.350  Understanding  the  gaps  in  current  data 
sources  and  determining  new  information  needs,  along  with  ways  to  gather  the  new 
information,  are  important  parts  of  a  performance  management  system.351 

A  review  of  data  quality  characteristics  among  agencies  and  subject-matter 
experts  is  represented  in  Figure  13.  This  illustration  shows  the  most  universally  included 

348  OMB,  Circular  No.  A-ll,  Part  6,  Section  260.9. 

349  Ibid. 

350  Ibid. 

351  Jackson,  The  Problem  of  Measuring  Emergency  Preparedness,  20-21. 
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elements  are  validity,  timeliness,  consistency,  accuracy,  and  ease  of  use.  The  GAO 
defines  these  terms  as: 

•  Validity — the  extent  to  which  the  data  adequately  represent  actual 
performance. 

•  Accuracy — the  extent  to  which  the  data  are  free  from  significant  error. 

•  Consistency — the  extent  to  which  data  are  collected  using  the  same 
procedures  and  definitions  across  collectors  and  times. 

•  Timeliness — whether  data  about  recent  perfonnance  are  available  when 
needed  to  improve  program  management  and  report  to  Congress. 

•  Ease  of  use — how  readily  intended  users  can  access  data,  aided  by  clear 
data  definitions,  user-friendly  software,  and  easily  used  access 
procedures.352 
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Figure  13.  Data  Quality  Characteristics 


352  Government  Accountability  Office  (GAO),  Perfonnance  Plans,  Selected  Approaches  for 
Verification  and  Validation  of  Agency’  Performance  Information  (GAO-GGD-99-139),  Washington,  DC: 
GAO,  July  1999,  13. 
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These  data  quality  elements  and  limitations  provide  the  framework  for  developing 
a  data  collection  plan.  The  collection  plan  should  indicate  how  often  each  measurement 
should  be  taken — hourly,  daily,  weekly,  monthly,  or  annually.  It  should  also  identify  the 
specific  date  and/or  time  of  day  that  will  provide  the  best  indicator  for  factors  affecting 
performance.  All  measurements  collected  should  have  an  associated  date/time  stamp  to 
facilitate  future  analysis. 

An  additional  consideration  in  establishing  the  collection  plan  is  who  is  going  to 
be  consuming  the  data,  measurements,  or  infonnation.  It  can  be: 

•  A  responsible  individual  for  the  program,  project  or  task 

•  The  individual  or  team  accountable  for  the  results 

•  Anyone  with  a  support  role  in  the  program  or  project  involved  in  the 
process 

•  Potentially  those  who  need  to  be  informed  of  program  performance  (to 
include  stakeholders  and  government  officials) 

Additional  questions  include:  What  is  the  best  format  in  which  to  present  the 
data?  What  are  the  best  methods  to  distribute  or  receive  feedback  on  the  data?  What 
analysis  or  manipulation  of  the  data  is  necessary  to  improve  usefulness  or  quality?  What 
costs  or  burden  in  the  collection  process  or  analysis  are  necessary  before  the  data  can  be 
useful?353  One  example  of  a  data  collection  plan  is  shown  in  Figure  14. 354 


353  OMB,  Circular  No.  A-ll,  Part  6,  Section  240.8. 

354  DHS,  Annual  Performance  Report;  Fiscal  Years  2013-2015,  53-54. 
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Performance 

Measure 

Percent  of  households  that  participated  in  a  prepared  ness  exercise  or  drill  at  their  workplace,  school,  home  or  other  community  location 
in  the  past  year  (New  Measure) 

Program 

Preparedness 

Description 

This  measure  calculates  the  percent  of  households  responding  to  a  survey  who  indicate  that  they  have  participated  in  a  preparedness 
exercise  or  drill  in  their  workplace,  school,  home,  or  community  in  the  past  year.  The  survey  collects  individual  disaster  preparedness 
data  from  a  random  sample  of  households  acrossthe  nation.  Improvingthe  public's  knowledge  and  ability  to  take  effective  protective 
actions  for  hazards  is  a  key  objective  of  preparing  the  public  to  act  quickly  and  effectively  in  emergency  situations. 

Scope  of  Data 

As  part  of  the  Nationwide  Household  Survey,  a  total  of  about  3,000  or  more  tele  phone  interviews  are  conducted  during  the  summer 
each  year  on  individual  and  household  preparedness.  The  survey  contacts  individuals  throughout  the  United  States  and  the  six  territories. 

Data  Source 

As  part  of  the  FEMA  National  Survey,  a  total  of  a  bout  3,000  or  more  telephone  interviews  are  conducted  yearly  on  individual  and 
household  prepared  ness.  The  survey,  which  is  conducted  by  National  Preparedness  Directorate  (NPD)  contractors,  collects  the  data  in 
the  statistical  analysis  program  SPSS  and  then  provides  a  reportto  NPD  on  the  survey  responses. 

Data  Collection 
Methodology 

The  measure  calculates  the  percent  of  households  surveyed  via  landline  or  cellular  phone  who  responded  affirmatively  to  the  question 
that  asked  whether  they  have  participated  in  a  disaster  preparedness  exercise  or  drill  in  their  workplace,  school,  home  or  another 
community  location  in  the  past  year.  Survey  data  is  collected  using  a  Computer  Assisted  Telephone  Interviewing  (CATI)  system  and 
results  from  the  survey  are  analyzed  in  SPSS  and  SAS.  When  processing  the  data  from  the  random  digit  dialing  surveys,  results  are 
weighted  to  correct  for  unequal  probabilities  of  selection.  The  sample  data  are  also  post-stratified  according  to  geography,  age,  gender 
and  race  to  account  for  potential  biases  such  as  over-and  under-representation  of  certain  population  segments.  This  will  adjust  the 
sample'sdemographicdistributionsto  match  the  distribution  derived  from  the  latest  available  Current  Population  Survey  estimates. 

Reliability  Index 

Reliable 

Explanation  of 

Data  Reliability 
Check 

There  is  currently  no  way  to  independently  verify  the  accuracy  of  participants’  responses  or  the  responses  recorded  by  the  survey 
administrator.  But,  each  programmed  survey  instrument  goes  through  a  rigorous  quality  control  process.  When  the  instrument  is  in  the 
field,  this  rigorous  quality  assurance  process  continues.  The  overall  process  includes,  but  is  not  limited  to,  program  testing,  a  pre-test  and 
cognitive  testing  to  determine  the  effectiveness  of  the  surveyand  questions,  monitoring  of  in-progress  calls,  recordingof  all  interviews, 
and  the  production  of  tab  ulations  of  every  question  and  variables  to  detect  any  missing  data  or  errors.  Additional  quality  measures 
include  the  checking  of  survey  skip  patterns  and  data  accuracy  and  consistency  checks. 

Figure  14.  DHS  Data  Collection  Plan 


Preparedness  metrics  “should  measure  the  ability  to  perform  the  missions  and 
tasks  assigned  to  them  by  the  strategy.  As  the  strategy  changes  and  evolves,  these  tasks 
can  change,  and  readiness  metrics  should  adapt  as  well.”355 

The  OMB  defines  a  target  as  “a  quantifiable  or  otherwise  measurable 
characteristic  that  tells  how  well  or  at  what  level  an  agency  or  one  of  its  components 
aspires  to  perform.  In  setting  and  communicating  targets,  agencies  should  include  the 
baseline  value  from  which  the  target  change  is  calculated.”356  DHS  uses  capability 
targets  in  describing  its  perfonnance  criteria. 

Capability  targets  are  defined  in  the  Goal  as  “the  perfonnance  threshold(s)  for 
each  core  capability.”  Further,  performances  measures  are  defined  as  “the  metrics  used  to 
ascertain  actual  performance  against  target  levels  identified  for  each  core  capability;  by 
design,  they  are  clear,  objective,  and  quantifiable.”357  Using  these  two  definitions,  it 
would  appear  the  Goal  intends  for  its  listed  capability  targets  to  begin  the  process  of 
designing  or  defining  a  perfonnance  measurement  system.  To  analyze  the  effectiveness 
of  measuring  performance  based  on  the  infonnation  provided  in  the  Goal,  it  is  useful  to 
analyze  a  sample  of  the  provided  capability  targets.  One  target  from  each  of  the  five  core 
capabilities  is  provided  with  a  brief  analysis  in  the  following  paragraphs.358 

Long-term  vulnerability  reduction  is  a  core  capability  within  the  mitigation 
mission  area.  One  of  the  targets  assigned  to  this  core  capability  is  to  “achieve  a 
measurable  decrease  in  the  long-term  vulnerability  of  the  Nation  against  current  baselines 
amid  a  growing  population  base  and  expanding  infrastructure  base.”  Some  of  the 
performance  measurement  challenges  in  in  this  area  are:  what  vulnerabilities  are 
detennined  a  priority  and  how  are  they  stratified;  how  the  current  vulnerability  baselines 
are  determined;  and  how  the  changing  population  dynamics  and  growing  infrastructure 
affect  the  collected  data’s  reliability. 


355  Harrison,  “Rethinking  Readiness,”  50. 

356  OMB,  Circular  No.  A-ll,  Part  6,  Section  200.21. 

357  FEMA,  National  Preparedness  Goal,  A-l,  A-2. 

358  Ibid.,  11-14. 
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Threats  and  hazard  identification  is  another  core  capability  within  the  mitigation 
mission  area.  One  of  the  listed  target  capabilities  is  to  “identify  the  threats  and  hazards 
within  and  across  the  states,  territories,  and  the  top  100  metropolitan  statistical  areas,  in 
collaboration  with  the  whole  community,  against  a  national  standard  based  on  sound 
science.”  The  annual  THIRA  report  completed  by  all  Homeland  Security  Grant  Program 
funding  is  a  key  component  in  meeting  this  target  capability.  However,  the  subjectivity  of 
the  entities  completing  the  report  raises  concerns;  How  can  sound  science  evaluate  the 
reported  threats  and  hazards,  and  how  can  the  national  standard  be  identified? 
Additionally,  the  level  of  whole  community  involvement  needs  to  be  better  defined  so  the 
proper  statistical  sample  population  can  be  included  in  the  individual  reports. 

Fatality  management  services  are  a  core  capability  within  the  response  mission 
area.  One  of  the  Goal’s  target  capabilities  is  to  “establish  and  maintain  operations  to 
recover  a  significant  number  of  fatalities  over  a  geographically  dispersed  area.”  To 
effectively  collect  measures  that  can  assess  this  capability,  standards  and  baseline 
information  need  to  be  established.  The  term  “significant  number”  also  needs  specificity 
to  eliminate  subjective  interpretation. 

Housing  is  a  core  capability  within  the  recovery  mission  area.  Its  target  capability 
is  to  “establish  a  resilient  and  sustainable  housing  market  that  meets  the  needs  of  the 
community,  including  the  need  for  accessible  housing  within  the  specified  time  frame  in 
the  recovery  plan.”  Again,  there  is  much  subjectivity  in  the  wording.  What  factors 
detennine  a  resilient  and  sustainable  housing  market,  and  how  would  those  factors 
change  based  on  the  multitude  of  economic,  geographic,  and  demographic  conditions  that 
occur  throughout  the  United  States?  What  standards  exist  for  the  jurisdictions  that  are 
establishing  the  time  frame  to  be  met? 

Intelligence  and  infonnation  sharing  is  a  core  capability  within  the  prevention 
mission  area.  A  listed  target  capability  is  to  “share  relevant,  timely,  and  actionable 
information  and  analysis  with  Federal,  state,  local,  private  sector,  and  international 
partners  and  develop  and  disseminate  appropriate  classified/unclassified  products.”  To 
collect  less  subjective  data,  there  are  many  tenns  in  this  target  area  that  need  specific 

definition,  including  relevant,  timely,  actionable,  and  appropriate. 
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The  target  capabilities,  as  identified  in  the  Goal,  provide  limited  utility  on  their 
own  merit.  Much  work  is  needed  to  meet  the  Goal’s  objective  to  “analyze  current 
performance  against  our  intended  capabilities,  the  defined  targets,  and  associated 
performance  measures.”359 

It  is  extremely  difficult  to  measure  or  otherwise  detennine  effectiveness  of  all  the 
identified  target  capabilities.  Newly  developed  quality  preparedness  metrics  will 
undoubtedly  not  be  perfect  predictors  of  actual  perfonnance,  and  may  not  measure  true 
capability.  The  aspiration,  then,  should  be  a  performance  measurement  system  that 
“comes  closer  to  collecting  relevant  performance  measures,  and  to  continue  improving 
and  refining  these  metrics  over  time.”360 

DHS  has  recognized  both  the  need  for  better  perfonnance  measures  and  the 
challenge  that  exists  in  collecting  them.  They  have  identified  a  new  initiative  called  the 
Analytic  Agenda  that  will  “enable  the  creation  of  a  full  set  of  perfonnance  measures  for  a 
given  study  topic;  support  analytically  infonned  strategy  development,  resource 
allocation,  investment,  and  operational  decision  making  for  that  topic;  facilitate 
systematic  program  evaluation;  and  optimally  deliver  indicator  and  warning  capabilities 
to  allow  the  Department  to  assume  an  anticipatory  posture.”361  Although  the  initiative’s 
focus  is  limited  by  leadership’s  priority  for  a  given  study  topic,  this  proposal  offers  hope 
for  the  inclusion  of  preparedness  measures  and  whole  community  involvement. 

“Strategy-based  metrics  are  vital  because  they  connect  strategy,  assigned 
missions,  and  mission-essential  tasks  to  the  readiness  data  being  collected  and 
analyzed.”362  The  links  between  strategy,  mission,  and  task  drive  the  manager  to  think 
and  analyze  in  tenns  of  outputs.  The  manager  uses  the  defined  strategic  goals  (outputs)  to 
inform  budget  and  resource  allocation  decisions  (inputs).  But  before  a  manager  can 
adequately  assess  the  impact  resources  have  on  mission  performance,  an  organization 
must  develop  measures  that  can  document  their  ability  to  accomplish  assigned  tasks.  If 

359  Ibid.,  19. 

360  Harrison,  “Rethinking  Readiness,”  52. 

361  DHS,  Fiscal  Years  2014-2018  Strategic  Plan,  12. 

362  Harrison,  “Rethinking  Readiness,”  62. 
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the  organization’s  ability  to  measure  their  perfonnance  of  mission  tasks  cannot  be 
quantified,  then  it  is  not  possible  to  evaluate  the  program’s  inputs  and  processes.363 

DHS  should  develop  a  performance  management  process  that  considers  multiple 
perspectives  and  evidence  sources  to  understand  the  progress  made  on  each  strategic 
objective.  Progress  toward  achieving  individual  quantitative  perfonnance  goals  related  to 
the  strategic  objective  is  one  important  consideration,  but  is  not  solely  representative  of 
the  scope,  complexity,  or  external  factors  that  can  influence  program  results  and 
outcomes.  When  reviewing  progress  on  each  strategic  objective,  agencies  should,  at  a 
minimum,  consider: 

•  If  desired  changes  have  occurred  in  the  ultimate  outcomes  the  agency 
seeks  to  improve  and  whether  these  outcomes  are  directly  measureable  or 
must  be  assessed  through  proxies  or  other  means  of  evaluation 

•  Progress  made  by  the  agency  toward  the  performance  goals  established  in 
the  most  recent  Annual  Performance  Plan  that  relate  to  the  strategic 
objective,  including  both  outcome  indicators  and  output  indicators 

•  Program  evaluations,  research  studies,  data  and  policy  analysis  or  other 
assessments  relevant  to  the  strategic  objective  or  the  related  programs 

•  External  factors  affecting  the  strategic  objective,  including  existing  and 
likely  changes  in  the  operating  environment,  the  size  of  program  demand, 
or  challenges  faced  during  program  execution 

•  Benchmarking  information  from  others  trying  to  accomplish  the  same  or 
similar  objectives  or  using  the  same  or  similar  key  process 

•  Lessons  learned  from  past  efforts  to  continuously  improve  service  delivery 
and  resolve  management  challenges,  especially  in  coordinating  across 
organization  components  and  with  delivery  partners 

•  Effectiveness  of  coordination  and  collaboration  across  organizational 
boundaries  and  with  delivery  partners  including  management  milestones 
met 

•  Identification,  assessment,  and  prioritization  of  probable  risks  that  may 
impact  program  delivery  or  outcomes  significantly  in  the  coming  year  or 
two 

•  Effectiveness  of  scaling  efforts 


363  Ibid.,  51. 
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•  Budgetary,  regulatory,  or  legislative  constraints  that  may  have  an  impact 
on  progress364 

G.  INFORMATION  TECHNOLOGY  SYSTEM 

In  DHS’s  2014-2018  Strategic  Plan,  the  Analytic  Agenda  initiative  described  the 
data  collection  and  management  challenge: 

The  Department  of  Homeland  Security  must  be  able  to  harness  vast 
amounts  of  data  to  infonn  strategy  and  future  planning.  There  are  a 
number  of  key  areas  where  DHS  must  improve  its  ability  to  collect  new 
data,  analyze  existing  data,  and  present  data  in  a  compelling  way  to  our 
partners  and  the  public.  The  Department’s  four-year  Analytic  Agenda 
provides  the  foundation  for  tackling  this  “Big  Data”  challenge  and 
supporting  analytically-informed  decision-making  across  DHS 
missions.365 

This  “big  data”  challenge  is  immense  given  the  Department’s  diverse  and 
complex  mission  sets.  Typically,  the  big  data  moniker  invokes  a  massive  infonnation 
technology  and  knowledge  management  system  acquisition.  DHS’s  priorities  are  stated  to 
be  “building  and  institutionalizing  the  necessary  data,  models,  and  business  process.”  The 
topical  development  approach  will  help  place  the  priorities  in  the  proper  perspective. 
Before  another  information  management  system  can  be  developed,  the  processes  and  data 
requirements  will  need  to  be  better  defined. 

The  DOD  has  used  the  Status  of  Readiness  and  Training  System  (SORTS)  since 
1986  to  “report  on  readiness  across  the  services.  SORTS  compares  the  level  of  inputs  to 
target  amounts  detennined  by  the  services.  Individual  units  are  measured  on  a  scale  of 
one  to  four  in  four  areas:  personnel,  equipment  and  supplies  on  hand,  equipment 
condition,  and  training.”366 

SORTS  uses  quantitative  indicators  and  commanding  officers’  judgments  to 
assess  if  units  have  the  personnel,  equipment,  training,  and  supplies  they  need  to  go  to 


364  OMB,  Circular  No.  A-ll,  Part  6,  Section  270.10. 

365  DHS,  Fiscal  Years  2014-2018  Strategic  Plan,  11. 

366  Harrison,  “Rethinking  Readiness,”  44. 
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war.367  While  SORTS  has  been  the  most  comprehensive  and  widely  used  readiness 
reporting  system  in  the  DOD  for  more  than  25  years,  it  has  also  been  subject  to  much  of 
the  same  criticism  attributed  to  DHS  measurement  systems.  Among  the  critiques  are:  it 
uses  subjective  inputs,  is  prone  to  errors,  lacks  standardization,  and  delivers  misleading 
scores  due  to  broad  measurement  standards  that  can  mask  underlying  problems  in  critical 
areas.368  The  DOD’s  newest  capability  system  is  the  Readiness  and  Reporting  System 
(DRRS).  Although  it  has  taken  more  than  10  years  to  implement,  it  is  now  widely 
accepted  in  DOD  as  an  improvement  over  SORTS.  Even  with  a  legacy  system  to  upon 
which  improve,  and  years  of  development,  two  concerns  about  DRRS  remain 
standardization  and  subjectivity.369 

The  USAF  utilizes  a  web-based  program  called  the  Management  Internal  Control 
Toolset  (MICT)  to  manage  the  enterprise  self-assessment  and  inspection  system.  It 
provides  information  transparency  and  data  ranging  from  a  complete  compilation  of  all 
current  and  relevant  guidance,  policy,  directives,  and  statutes  to  audit  histories  and 
individual  self-assessment  results.  The  USAF  also  uses  the  self-assessment  data  and 
history  to  more  effectively  manage  their  unit  inspections  by  identifying  specific  subject- 
matter  experts  for  on-site  inspections  to  ensure  the  identified  problems  and  challenges  are 
fully  addressed.  These  experts  can  also  provide  mentoring  and  assistance  in  resolving  the 
issues.370  Finally,  when  the  self-assessment  identifies  a  program  or  process  deficiency, 
MICT  has  a  built-in  tool  to  facilitate  a  corrective  action  process  that  tracks  progress  until 
the  root  cause  of  the  shortfall  is  under  control.371  MICT’s  standardized  and 
comprehensive  data  storage  can  also  benefit  headquarters  staff  through  enterprise-wide 


367  Adebayo  Adedeji,  Linking  the  Readiness  of  the  Armed  Forces  to  DOD ’s  Operation  and 
Maintenance  Spending,  (Washington,  DC:  Congressional  Budget  Office,  2011), 
https://www.cbo.gov/sites/default/files/04-25-readiness.pdf. 

368  R.  Derek  Trunkey,  Implications  of  the  Department  of  Defense  Readiness  Reporting  System 
(Washington,  DC:  Congressional  Budget  Office,  2013),  Working  Paper  2013-03,  6. 

369  Trunkey,  Implications  of  the  Department  of  Defense  Readiness  Reporting  System,  12. 

370  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  29. 

371  Camm  et  at,  Charting  the  Course,  xv. 
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trend  analysis  and  can  reveal  systemic  issues  that  warrant  close  or  immediate  attention.  It 
can  also  more  effectively  identify  best  practices.372 

The  DOD  has  used  readiness  and  capability  reporting  systems  for  decades,  and 
should  be  considered  for  lessons  learned  and  benchmarking.  The  military  and  homeland 
security  enterprise  have  common  missions;  both  respond  to  crisis  situations;  prepare  for 
the  unexpected  through  training  and  exercises;  acquire,  maintain,  and  sustain  professional 
equipment;  and  maintain  a  cadre  of  experienced  and  ready  personnel.  These  shared  traits 
provide  the  basis  to  explore  the  DOD’s  system  of  assessment  for  features  that  will 
mitigate  long-standing  perfonnance  measurement  issues  in  DHS. 


372  Ibid.,  78. 
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V.  CONCLUSIONS 


The  conclusions  and  recommendations  from  this  research  form  a  starting  point  to 
better  understand  preparedness  within  the  homeland  security  enterprise,  and  to  develop  a 
means  to  accurately  report  and  understand  national  gaps  and  priorities. 

There  are  many  established  tools  and  techniques  to  apply  to  the  perfonnance 
management  system.  A  few  of  these  were  discussed  in  this  thesis  to  illustrate  potential 
applications,  but  the  discussion  only  serves  as  an  introduction  to  what  will  be  a  long  and 
intensive  effort.  Success  in  this  endeavor  will  only  be  realized  through  persistent  and 
dedicated  efforts  with  support  and  engagement  from  senior  leaders  across  the  enterprise. 

A.  FINDINGS 

The  Homeland  Security  Enterprise  Lacks  Sufficient  Performance 

Management  Expertise 

While  federal  agencies  are  required  to  have  performance  improvement  officers 
(PIO)  included  within  their  organizations,  these  professionals’  ability  to  adequately 
address  process  performance  across  the  whole  of  community  is  inadequate.  If  reporting 
on  national  preparedness  is  to  improve,  a  dedicated  and  knowledgeable  team  of 
performance  professionals  working  within  the  SLTT  jurisdictions  is  needed. 

(1)  Recommendation  1:  DHS  should  establish  an  initial  cadre  of  SLTT 
performance  improvement  officers  providing  one  position  to  each  of  the 
56  jurisdictions. 

Creating  federal  PIO  positions  will  ensure  standardization  and  control  over  the 
process.  This  is  essential  to  reduce  the  tendency  for  local  jurisdictions  to  use  the 
performance  management  process  improperly  or  skew  reported  data  to  gain  an  advantage 
in  grant-funding  allocations.  Additionally,  hiring  the  PIOs  under  federal  systems  will 
establish  benchmarks  for  the  required  PIO  knowledge,  skills,  and  abilities  and 
standardize  their  salary  schedules  to  provide  uniform  competency  across  the  enterprise. 

The  portfolio  of  these  positions  would  include  all  of  the  DHS-related  block  grant 
funding  streams.  Therefore,  states  with  more  complex  federal  revenue  streams  due  to 
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multiple  UASI  cities,  water  ports,  and  tribal  entities,  among  other  grant  programs,  will 
require  a  potentially  higher-experienced  PIO,  or  even  perhaps  an  assistant.  These 
additional  requirements  should  be  balanced  by  the  less  complex  states  and  territories.  If 
the  average  salary  for  a  PIO  position  is  $100,000,  and  this  figure  is  tripled  to  account  for 
benefits  and  job  function  expenses,  the  total  annual  outlay  for  sustaining  the  human 
capital  portion  of  this  capability  is  approximately  $16.8  million. 

The  amount  provided  to  SLTT  jurisdictions  through  FY  2014  preparedness  grant 
programs  was  in  excess  of  $  1 .6  billion,  making  the  investment  in  performance  capability 
only  0.01  percent  of  the  total  allocation.  Prior  analysis  indicated  “the  cost  of  using  federal 
employees  or  contractors  to  collect  and  validate  preparedness  data  at  the  state  and  local 
levels  would  be  cost-prohibitive.”373  Using  these  resources  to  analyze  and  inform  the 
entire  preparedness  process — not  just  collecting  and  validating  data — should  easily  pay 
for  itself  through  increased  efficiencies  and  a  more  effective  national  program. 

Undoubtedly,  this  recommendation  will  not  resolve  DHS’s  performance 
management  challenges  alone.  The  SLTT  jurisdictions  will  also  need  to  train  executives 
and  program  managers  in  performance  improvement  techniques  and  processes.  SLTT 
jurisdictions  should  also  dedicate  a  staff  position  to  work  with  the  federal  PIO. 

(2)  Recommendation  2:  DHS  should  establish  standardize  performance 
management  training  curricula  or  identify  an  existing  training  program  for 
federal  and  SLTT  personnel. 

Creating  a  training  program  specific  to  DHS  will  result  in  standardize  tools  and 
techniques  used  throughout  the  enterprise.  It  will  also  serve  as  a  means  to  capture  lessons 
learned  and  good  practices  from  the  field  as  PIO  practitioners  move  through  advanced 
courses  of  study.  A  DHS-specific  training  program  can  also  be  tailored  to  provide 
awareness  training  for  operational  staff  to  better  understand  their  roles  in  the  perfonnance 
management  process  as  well  as  the  importance  of  standardized  measurement  collection 
and  reporting. 


373  GAO,  FEMA  Has  Made  Limited  Progress  in  Efforts  to  Develop  and  Implement  a  System,  13. 
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Performance  Measures  for  the  Enterprise  are  Ineffective  and 

Deficient 

The  cadre  of  PIOs  focused  on  SLTT  performance  management  using  standardized 
methodologies  will  provide  the  best  opportunity  to  identify  valid,  timely,  consistent, 
attainable,  and  actionable  data  on  SLTT  programs  and  processes  that  will  better  inform 
DHS  and  Congress  on  national  preparedness.  Over  time,  these  efforts  will  help  inform 
funding  levels  and  priorities  for  national  priorities.  In  addition,  the  data  will  begin  to 
show  program  development  progress  and  promote  efficiency  by  identifying  sustainment 
funds  for  mature  programs  and  freeing  up  resources  for  building  the  next  priority 
capability. 

In  conjunction  with  the  development  of  perfonnance  metrics,  the  PIO  cadre  will 
be  invested  in  designing  and  contributing  to  the  state  and  national  preparedness  reports, 
state  and  federal  strategic  planning  efforts,  THIRA  reporting,  grant  reporting 
requirements,  and  federal  and  state  process  improvement  efforts. 

(3)  Recommendation  3:  DHS  should  establish  the  critical  few  common 
measures  within  each  mission  area  and  initial  reporting  criteria  for  SLTT 
jurisdictions. 

DHS,  through  the  preparedness  PIOs,  should  identify  two  to  three  common 
metrics  for  each  of  the  five  DHS  mission  areas  for  SLTT  jurisdictions  to  collect  and 
analyze.  Each  metric  will  have  established  standards  and  criteria.  Once  enough  data  is 
collected  to  analyze  and  establish  trends,  the  data  will  be  evaluated  for  quality  and 
relevance.  Acceptable  data  elements  will  be  incorporated  into  the  perfonnance 
management  system  with  benchmarked  performance  goals  and  timelines. 

(4)  Recommendation  4:  DHS  should  delay  investment  in  and  development 
of  an  SLTT -specific  perfonnance/comprehensive  assessment  system. 

Any  further  investment  into  developing  a  web-based  or  other  comprehensive 

assessment  system  should  be  postponed  until  the  metrics  feeding  the  perfonnance 

management  system  are  vetted  and  functional,  and  the  process  for  reporting  and 

interfacing  with  the  system  are  established  and  tested.  Incorporating  SLTT  data  into  the 

system  will  significantly  change  how  information  and  knowledge  are  generated  and 
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increase  the  complexity  of  analysis.  Understanding  the  new  process  and  infonnation 
availability  will  inform  the  technological  system  requirements.  Additional  processes  to 
consider  including  in  the  next  version  of  the  system  are:  audit,  assessment,  and  inspection 
findings;  after-action  reports;  and  corrective  action  plans  and  results. 

The  next  iteration  of  the  comprehensive  assessment  system  should  take  advantage 
of  the  capabilities  of  legacy  systems  as  well  as  the  lessons  learned  and  shortfalls.  Further 
development  of  a  perfonnance  assessment  system  includes  the  “need  for  more  effective 
coordination  with  state,  local,  and  tribal  governments  and  the  need  for  infonnation 
technology  systems  that  are  updated  and  integrated  throughout  the  agency.”374 

(5)  Recommendation  5:  The  DHS  should  develop  measures  of  efficiency  and 
effectiveness  for  the  PIO  program  to  identify  areas  for  improvement  and 
report  on  value  for  savings  achieved. 

SLTT  PIO  program  will  need  to  show  value  and  results  for  continued  funding  and 
ultimate  sustainment.  These  measures  should  undergo  the  same  rigor  and  quality 
standards  as  the  key  metrics  supporting  the  core  mission  areas.  This  data  should  inform 
senior  leaders  on  effective  use  of  the  resources  and  highlight  areas  for  improvement. 

Alignment  of  Strategic  Goals  Is  Weak 

DHS’s  ability  to  assess  and  report  on  national  preparedness  is  dependent  on  SLTT  capability 
reporting.  To  gain  a  true  assessment,  data  from  the  private  sector,  non-governmental  organizations,  and 
non-profit  organizations  will  also  need  to  be  collected  and  included  in  the  overall  assessment.  The  process 
of  collecting  and  compiling  this  data,  however,  will  require  an  iterative  approach  that  includes  adding 
additional  sources  of  data  as  the  system  matures.  Aligning  the  data  collection  efforts  of  the  SLTT 
jurisdictions  to  national  strategic  priorities  will  facilitate  a  more  comprehensive  and  accurate  national 
preparedness  assessment. 

(6)  Recommendation  6:  DHS  should  devise  a  method  to  normalize  data  that 
does  not  share  common  attributes  or  collection  standards. 

Data  that  is  standardized  and  common  across  the  entire  homeland  security 
enterprise  will  be  small  in  comparison  to  the  total  available  data.  To  adequately  assess 

374  DHS  Office  of  the  Inspector  General,  OIG  10-23,  FEMA’s  Preparedness  for  the  Next 
Catastrophic  Disaster  -An  Update,  September  2010,  8. 
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national  preparedness,  the  non-standard  data  or  data  unique  to  specific  jurisdictions  will 
need  to  be  developed  in  such  a  way  that  it  can  contribute  to  the  overall  assessment.  Data 
quality  will  continue  to  be  a  significant  concern  to  ensure  the  data  input  retains  validity, 
consistency,  timeliness,  and  accuracy.  Nonnalization  of  this  data  can  be  accomplished 
using  benchmarked. 

(7)  Recommendation  7:  DHS  should  use  OIG  and  other  external  assessment 
findings  to  detennine  challenges  and  issues  systemic  to  the  homeland 
security  enterprise. 

The  OIG  has  perfonned  audits  and  assessments  of  state  and  local  homeland 
security  programs  for  many  years,  reporting  on  findings  in  areas  such  as  “homeland 
security  strategies,  obligation  of  grants,  reimbursement  to  subgrantees  for  expenditures, 
and  monitoring  of  subgrantees ’  perfonnance  and  financial  management,  procurement, 
and  property  management.”375  Many  of  the  OIG’s  findings  have  been  reported  on  over 
many  years  and  other  findings  are  evident  in  almost  all  of  the  jurisdictions  assessed. 
Processes  and  procedures  deemed  critical  to  the  performance  of  the  homeland  security 
mission  or  proper  stewardship  of  taxpayer  dollars  are  as  important  to  the  overall 
enterprise  as  building  capabilities.  These  identified  findings  should  become  a  focus  of  the 
performance  management  system  and  should  be  included  in  metric  development  and 
tracked  to  resolution. 

A  technique  to  analyze  OIG  data  can  be  derived  from  the  USAF  OIG.  They 
establish  broad  categories  of  codes  to  be  used  in  identifying  deficiencies,  results,  or 
findings  during  their  inspection  process.  The  codes  are  broken  down  into:  equipment/ 
tools,  guidance,  leadership/supervision  (LS),  resource  shortfall  (RS),  safety,  training,  and 
human  factors.  In  addition,  sub-categories  are  assigned  with  a  sequential  numbering 
system.  For  example,  “LS8 — Unit  failed  to  adequately  program  resources”  is  a  code 
given  when  management  improperly  allocated  resources.  In  this  instance,  the  causal 
factor  was  attributed  to  poor  management  decisions  and  not  to  a  general  lack  of  resources 


375  Department  of  Homeland  Security  Office  of  Inspector  General,  Countering  Terrorism;  Semi- 
Annual  Report  to  the  Congress  (Washington,  DC:  DHS,  April  1,  2014 — September  30,  2014), 
https://www.oig.dhs.gov/assets/SAR/OIG_SAR_Aprl4_Sepl4.pdf. 
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where  a  code  of  RS  would  have  been  assigned.376  This  type  of  coding  system  facilitates 
the  analysis  of  data  through  categorization  and  provides  the  headquarters  agency  a  means 
to  identify  trends  across  the  enterprise.  This  analysis  can  highlight  areas  for  future 
analysis  and  expose  gaps  in  data  and  infonnation. 

(8)  Recommendation  8:  DHS  should  expand  the  SLTT  PIO  role  to  include 
all  federal  block  grant  programs. 

The  focus  of  this  thesis  was  limited  to  national  preparedness  efforts  within  the 
DHS  mission  portfolio.  Because  of  the  broad  scope  of  interdependencies  within  the 
homeland  security  enterprise,  there  are  numerous  opportunities  for  other  federal 
stakeholders  to  include  their  SLTT  grant  programs  in  the  performance  management 
system.  Stakeholders  such  as  the  Departments  of  Justice,  Interior,  Treasury,  Housing  and 
Urban  Development,  Agriculture,  Commerce,  Education,  Defense,  State,  Health  and 
Human  Services,  and  Transportation  all  have  SLTT  grant  programs  that  either 
complement  or  directly  support  DHS  missions  and  goals.  Analyzing  all  of  the  grant 
opportunities  and  aligning  the  outcomes  of  individual  grant  goals  and  objectives  to 
national  priorities  would  reduce  redundancy  of  effort  and  improve  overall  effectiveness. 
Improved  cooperation  and  coordination  among  the  programs  at  the  federal  level  will  also 
facilitate  improved  synergy  among  the  associated  departments  within  and  among  the 
SLTT  jurisdictions. 

The  Self-assessment  and  Reporting  Process  Is  Ineffective 

The  primary  weakness  of  prior  DHS  assessment  systems  was  the  inconsistencies  generated  by 
self-assessing  and  reporting.  While  self-assessment  will  always  be  a  component  of  the  overall  evaluation,  a 
means  to  validate  the  reported  information  is  needed.  The  addition  of  the  SLTT  PIOs  provides  a  means  to 
validate  select  processes  and  data  collection  methods  on  a  random  and  as-required  basis.  A  rigorous  and 
comprehensive  validation  and  review  will  be  required  within  a  predetermined  period  to  ensure  strategic 
decisions  are  based  on  reliable  data. 

(9)  Recommendation  9:  DHS  should  expand  the  OIG’s  role  to  include  an 
assessment  of  perfonnance  management  and  business  processes. 


376  Secretary  of  Air  Force  (IG),  Air  Force  Inspection  System,  214-220. 
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Current  assessments  by  DHS  OIG  or  DHS  headquarters  staff  tend  to  evaluate 
SLTT  products  such  as  strategic  plans,  THIRA  data,  or  preparedness  reports  as  complete 
or  incomplete.  This  approach  does  not  evaluate  how  these  documents  are  integrated  into 
the  SLTT  decision-making  process,  stakeholders’  involvement  in  generating  the 
documents,  or  the  rigor  and  analysis  put  into  the  products.  The  result  is  a  check-box 
mentality  that  overlooks  or  fails  to  take  full  advantage  of  the  process’s  benefits.  An 
expanded  OIG  assessment  into  the  agency’s  business  practices  would  provide  increased 
confidence  in  data  quality  and  local  capabilities. 

(10)  Recommendation  10:  DHS  should  incorporate  practitioners  from  federal 
as  well  as  SLTT  jurisdictions  into  the  OIG  assessment  process. 

The  OIG  should  be  augmented  by  highly  experienced  and  specifically  trained 
staff  to  facilitate  the  expanded  assessment.  These  staff  should  be  vetted  by  central 
authority  for  required  experience  and  expertise,  and  nominated  as  OIG  inspection  team 
augmenters.  Once  approved,  the  staff  member  should  attend  an  inspection  training  course 
and  complete  a  set  number  of  inspections  as  an  observer  before  being  cleared  as  a 
primary  inspection  team  augmenter.  Augmenters  should  come  from  a  wide  range  of 
agencies  and  offices  to  include  DHS,  FEMA,  FEMA  regional  offices,  tribal,  state,  and 
local  jurisdictions.  This  practice  will  improve  information  sharing  and  distribution  of 
good  practices  across  the  enterprise. 

Policy  Decisions  on  Roles  and  Responsibilities  are  Needed 

National  preparedness  has  been  defined  through  the  whole  community  concept. 
Each  part  of  the  whole  community  needs  to  understand  its  function  in  the  enterprise  and 
its  expected  actions  to  best  satisfy  preparedness  requirements.  As  funding  allocation 
decisions  are  made  to  develop  and  carry  out  those  functions,  the  appropriate  level  of 
government  that  can  best  oversee  the  accomplishment  of  the  goals  and  objectives  needs 
to  be  identified.  Federal  and  state  agencies  need  to  be  involved  in  an  open  and  transparent 
debate  about  the  veracity  of  analyses  that  determine  funding  levels.  Only  then  will 
different  levels  of  government  understand  how  effectively  and  efficiently  they  have  used 
preparedness  funds. 
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(11)  Recommendation  11:  DHS  should  clearly  define  specific  roles  and 
responsibilities  for  developing  capabilities  within  each  mission  area. 

To  best  determine  funding  decisions,  every  agency  or  department  will  need  to 
understand  how  they  are  expected  to  contribute  to  meeting  national  strategic  goals  and 
objectives.  Without  a  clear  understanding,  SLTT  jurisdictions  could  either  squander 
resources  to  build  capabilities  that  are  already,  or  SLTT  jurisdictions  could  erroneously 
assume  the  capability  is  being  managed  by  the  federal  government,  resulting  in  a  gap  in 
national  capability. 

Funding  Decisions  Are  Required 

Funding  decisions  will  drive  DHS’s  ultimate  ability  to  successfully  report  on  the 
state  of  national  preparedness.  However,  the  combination  of  recommendations  presented 
here  goes  beyond  satisfying  a  mandated  report.  In  addition  to  better  understanding 
current  national  preparedness  levels,  these  recommendations  serve  as  a  foundation  to 
improve  the  enterprise  through  a  more  efficient  use  of  resources  and  through  better- 
informed  decisions  making  and  priority  setting. 

Establishing  a  cadre  of  56  PIOs  to  facilitate  SLTT  perfonnance  assessment  (and 
general  process  improvement)  is  not  the  total  solution.  There  will  be  costs  for  an 
increased  oversight  and  assessment  program  as  well  as  specific  training  programs.  These 
costs  are  essential  to  begin  building  the  capability  to  meet  the  intent  of  GPRAMA  and 
other  federal  laws. 

(12)  Recommendation  12:  DHS  should  fund  PIO  capability  for  all  56  states 
and  territories  and  an  expansion  of  SLTT  oversight. 

This  funding  should  remain  stable  over  a  five-to-ten-year  period  for  program 
stabilization.  The  impact  to  SLTT  grant  recipients  should  not  dramatically  change  until 
some  maturity  in  the  data  collection  process  is  developed  and  there  is  a  reasonable 
confidence  in  the  analysis. 
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B.  LIMITATIONS 

1.  Political  Pressures 

One  of  the  core  challenges  of  a  transparent  and  data-rich  decision-making 
environment  for  detennining  funding  allocation  is  the  resulting  change  to  the  status  quo. 
Jurisdictions  that  have  influential  or  powerful  congressional  delegations  could  feel 
compelled  to  intervene  when  their  constituents  receive  fewer  funds  as  a  result  of  the  data 
analysis.  Arguments  could  be  made  that  the  data  is  inaccurate  or  incomplete,  or  that  the 
analysis  was  flawed  or  skewed.  These  interventions  could  undermine  the  entire  effort  of 
maturing  the  performance  management  system.  Achieving  consistent  measures  of 
effectiveness  over  time  is  difficult  due  to  the  demands  of  an  ever-changing  market 
environment  and  goal  expectations.  This  is  especially  true  for  disaster  management 
agencies,  whose  goals  may  be  affected  by  the  vagaries  of  political  conditions  rather  than 
market  conditions.377 

2.  Investment  of  Time  and  Resources 

Just  as  Congress  has  demanded  an  accounting  for  the  money  spent  in 
preparedness  grants,  so,  too,  will  DHS  want  to  see  a  positive  return  on  investment  for  the 
cadre  of  PIOs.  DHS  should  establish  meaningful  goals  and  objectives  and  a  realistic 
timeline  for  developing  the  PIO  program.  In  addition,  a  robust  communication  plan  and 
awareness  campaign  should  be  initiated  for  appropriate  congressional  committees  and 
stakeholders  with  frequent  updates. 

3.  Leadership  Buy-in 

Program  success  will  require  support  from  the  entire  enterprise.  The  assigned 
PIOs  will  provide  the  necessary  skills  and  knowledge  to  the  perfonnance  management 
process,  but  results  will  depend  upon  SLTT  leaders,  directors,  and  program  managers’ 
abilities  to  implement  and  sustain  an  effective  system. 


377  Kirschenbaum,  “Measuring  the  Effectiveness  of  Disaster  Management  Organizations,”  81. 
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4.  Enterprise  Instability 

Emergencies  and  disasters  are  often  “all  hands  on  deck”  events  that  take  SLTT 
staff  members  away  from  their  normal  duties.  The  time  to  attend  to  response  and 
recovery  efforts  for  large  events  (relative  to  the  jurisdictions’  capacity  and  experience) 
can  disrupt  normal  operations  for  weeks  or  months.  During  this  time,  the  priority  for 
resource  utilization  will  likely  not  include  collecting  and  analyzing  performance  data. 
This  poses  a  dilemma  for  the  performance  management  system,  since  some  of  the  most 
valuable  outcome  and  output  data  is  exhibited  during  crisis  events.  Incorporating  data 
collection  into  the  nonnal  incident  command  tools  and  techniques  will  facilitate  the 
collection  and  analysis  of  valuable  data  without  detracting  from  nonnal  response  and 
recovery  activities. 

C.  SUMMARY 

DHS  is  an  agency  accustomed  to  wicked  problems.  Indeed,  an  array  of  wicked 
problems  is  at  the  essence  of  its  core  missions,  and  even  inherent  in  DHS’s  founding 
motivation.  Perfonnance  management  is  not  one  of  DHS’s  core  mission  areas,  but  is  a 
discipline  that  can  help  focus  all  of  its  core  missions  and  transfonn  the  homeland  security 
enterprise  into  a  more  effective  collaboration  with  measurable  efficiency.  DHS’s  inability 
to  adequately  report  on  perfonnance  indicators  over  the  past  decade  has  elevated  its 
performance  management  efforts  to  a  “wicked”  status. 

For  years,  DHS  implemented  top-down  strategies  and  approaches  to  establish  a 
preparedness  performance  management  system.  As  of  this  writing,  there  are  few 
substantive  results  to  show  for  the  effort.  The  recommendations  proposed  in  this  thesis 
will  shift  the  primary  emphasis  and  effort  of  collecting  and  analyzing  preparedness  data 
from  DHS  headquarters  to  the  SLTT  grant  recipients.  Utilizing  federal  PIOs  to  guide  and 
oversee  the  process  will  facilitate  a  standardized  approach,  while  utilizing  SLTT 
expertise  for  capturing  data,  identifying  innovative  strategies,  and  initiating  collaborative 
solutions. 
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